Non-touch optical detection of vital signs from amplified visual variations of reduced images of skin

ABSTRACT

A microprocessor is operably coupled to a camera from which patient vital signs of reduced images of skin are determined. A temporal variation of reduced images of skin from the camera is generated from multiple filters and then amplified from which the patient vital sign, such as heart rate or respiratory rate, can be determined and then displayed or stored.

RELATED APPLICATIONS

This application claims the benefit and priority under 35 U.S.C. 119 toGB1411983.8 filed on 4 Jul. 2014.

This application is a continuation of, and claims the benefit andpriority under 35 U.S.C. 120 of U.S. Original application Ser. No.14/324,235 filed on 6 Jul. 2014.

FIELD

This disclosure relates generally to motion amplification in images.

BACKGROUND

Conventional personal computers implement motion amplification.

BRIEF DESCRIPTION

One aspect includes an apparatus of motion amplification to communicatebiological vital signs, the apparatus including: a cropper that isoperable to receive at least two images and crop the image to exclude aborder area of the images, generating at least two cropped images, askin-pixel-identifier that is operably coupled to the cropper and thatidentifies pixel values that are representative of the skin in at leasttwo cropped images, a first frequency filter that is operably coupled tothe skin-pixel-identifier and that applies a frequency filter to outputof the skin-pixel-identifier, a regional facial clusterial module thatis operably coupled to the first frequency filter and that appliesspatial clustering to the output of the first frequency filter, a secondfrequency filter that is operably regional facial clusterial module andthat is applied to output of the regional facial clusterial module,generating a temporal variation, a vital-sign generator that is operablycoupled to the second frequency filter that generates at least one vitalsign from the temporal variation, and a display device that is operablycoupled to the vital-sign generator that is operable to display the atleast one vital sign.

One aspect includes an apparatus of motion amplification to communicatebiological vital signs, the apparatus including: a cropper that isoperable to receive at least two images and operable to crop the imagesto exclude a border area of the images, generating at least two croppedimages, a first frequency filter that applies a frequency filter to theat least two cropped images, a regional facial clusterial module that isoperably coupled to the first frequency filter and that applies spatialclustering to output of the first frequency filter, a second frequencyfilter that is operably to the regional facial clusterial module andthat is applied to output of the regional facial clusterial module,generating a temporal variation, a vital-sign generator that is operablycoupled to the second frequency filter that generates at least one vitalsign from the temporal variation, and a display device that is operablycoupled to the vital-sign generator that is operable to display the atleast one vital sign.

Apparatus, systems, and methods of varying scope are described herein.In addition to the aspects and advantages described in this summary,further aspects and advantages will become apparent by reference to thedrawings and by reading the detailed description that follows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a non-touch thermometer that does notinclude a digital infrared sensor, according to an implementation;

FIG. 2 is a block diagram of a non-touch thermometer that does notinclude an analog-to-digital converter, according to an implementation;

FIG. 3 is a block diagram of a non-touch thermometer having a colordisplay device, according to an implementation;

FIG. 4 is a flowchart of a method to determine a temperature from adigital infrared sensor, according to an implementation;

FIG. 5 is a flowchart of a method to display temperature colorindicators, according to an implementation of three colors;

FIG. 6 is a flowchart of a method to manage power in a non-touchthermometer having a digital infrared sensor, according to animplementation;

FIG. 7 is a block diagram of an apparatus of motion amplification,according to an implementation.

FIG. 8 is a block diagram of an apparatus of motion amplification,according to an implementation.

FIG. 9 is a block diagram of an apparatus of motion amplification,according to an implementation.

FIG. 10 is a block diagram of an apparatus of motion amplification,according to an implementation.

FIG. 11 is a block diagram of an apparatus of motion amplification,according to an implementation;

FIG. 12 is a block diagram of an apparatus to generate and present anyone of a number of biological vital signs from amplified motion,according to an implementation;

FIG. 13 is a block diagram of an apparatus of motion amplification,according to an implementation;

FIG. 14 is a block diagram of an apparatus of motion amplification,according to an implementation;

FIG. 15 is an apparatus that performs motion amplification to generatebiological vital signs, according to an implementation;

FIG. 16 is a flowchart of a method of motion amplification, according toan implementation;

FIG. 17 is a flowchart of a method of motion amplification, according toan implementation that does not include a separate action of determininga temporal variation;

FIG. 18 is a flowchart of a method of motion amplification, according toan implementation;

FIG. 19 is a flowchart of a method of motion amplification, according toan implementation;

FIG. 20 is a flowchart of a method of motion amplification from which togenerate and communicate biological vital signs, according to animplementation;

FIG. 21 is a block diagram of a hand-held device, according to animplementation;

FIG. 22 illustrates an example of a computer environment, according toan implementation;

FIG. 23 is a representation of display that is presented on the displaydevice of apparatus in FIGS. 1-3 and 24-28, according to animplementation;

FIG. 24 is a portion of a schematic of a circuit board of a non-touchthermometer, according to an implementation;

FIG. 25 is a portion of the schematic of the non-touch thermometerhaving the digital IR sensor, according to an implementation;

FIG. 26 is a portion of the schematic of the non-touch thermometerhaving the digital IR sensor, according to an implementation;

FIG. 27 is a circuit that is a portion of the schematic of the non-touchthermometer having the digital IR sensor, according to animplementation; and

FIG. 28 is a circuit that is a portion of the schematic of the non-touchthermometer having the digital IR sensor, according to animplementation.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings that form a part hereof, and in which is shown byway of illustration specific implementations which may be practiced.These implementations are described in sufficient detail to enable thoseskilled in the art to practice the implementations, and it is to beunderstood that other implementations may be utilized and that logical,mechanical, electrical and other changes may be made without departingfrom the scope of the implementations. The following detaileddescription is, therefore, not to be taken in a limiting sense.

The detailed description is divided into four sections. In the firstsection, apparatus of digital infrared sensor implementations aredescribed. In the second section, implementations of methods of digitalinfrared sensors are described. In the third section, implementations ofapparatus of vital sign amplification are described. In the fourthsection, implementations of methods of vital sign amplification aredescribed. In the fifth section, hardware and operating environments inconjunction with which implementations may be practiced are described.Finally, in the sixth section, a conclusion of the detailed descriptionis provided. The apparatus and methods disclosed in the third and fourthsections are notably beneficial in generating a temporal variation fromwhich a heartrate and the respiratory rate can be generated.

Digital Infrared Sensor Apparatus Implementations

In this section, particular apparatus of implementations are describedby reference to a series of diagrams.

FIG. 1 is a block diagram of a non-touch thermometer 100 that does notinclude a digital infrared sensor, according to an implementation.Non-touch thermometer 100 is an apparatus to measure temperature. Thenon-touch thermometer 100 includes a microprocessor 102. The non-touchthermometer 100 includes a battery 104 that is operably coupled to themicroprocessor 102. The non-touch thermometer 100 includes a singlebutton 106 that is operably coupled to the microprocessor 102. Thenon-touch thermometer 100 includes a digital infrared sensor 108 that isoperably coupled to the microprocessor 102. The digital infrared sensor108 includes digital ports 110 that provide only digital readout signal112. The non-touch thermometer 100 includes a display device 114 that isoperably coupled to the microprocessor 102. The microprocessor 102 isoperable to receive from the digital ports 110 that provide only digitalreadout signal 112. The digital readout signal 112 that isrepresentative of an infrared signal 116 detected by the digitalinfrared sensor 108. The microprocessor 102 is operable to determine thetemperature 120 from the digital readout signal 112 that isrepresentative of the infrared signal 116. The non-touch thermometer 100includes a camera 122 that is operably coupled to the microprocessor 102and is operable to provide two or more images 124 to the microprocessor102.

FIG. 2 is a block diagram of a non-touch thermometer 200 that does notinclude an analog-to-digital converter, according to an implementation.The non-touch thermometer 200 does not include an analog-to-digital(A/D) converter 202 operably coupled between the digital infrared sensor108 and the microprocessor 102. The digital infrared sensor 108 alsodoes not include analog readout ports 204. The dashed lines of theanalog-to-digital converter 202 and the analog readout ports 204indicates absence of the A/D converter 202 and the analog readout ports204 in the non-touch thermometer 200. The non-touch thermometer 200includes a microprocessor 102. The non-touch thermometer 200 includes abattery 104 that is operably coupled to the microprocessor 102. Thenon-touch thermometer 200 includes a single button 106 that is operablycoupled to the microprocessor 102. The non-touch thermometer 200includes a digital infrared sensor 108 that is operably coupled to themicroprocessor 102 with no analog-to-digital converter that is operablycoupled between the digital infrared sensor 108 and the microprocessor102, the digital infrared sensor 108 having only digital ports 110, thedigital infrared sensor 108 having no analog sensor readout ports. Thenon-touch thermometer 200 includes and a display device 114 that isoperably coupled to the microprocessor 102, where the microprocessor 102is operable to receive from the digital ports 110 a digital readoutsignal 112 that is representative of an infrared signal 116 detected bythe digital infrared sensor 108 and the microprocessor 102 is operableto determine the temperature 120 from the digital readout signal 112that is representative of the infrared signal 116. The non-touchthermometer 200 also includes a camera 122 that is operably coupled tothe microprocessor 102 and is operable to provide two or more images 124to the microprocessor 102.

In some implementations, the digital IR sensor 108 is a low noiseamplifier, 17-bit ADC and powerful DSP unit through which high accuracyand resolution of the thermometer is achieved.

In some implementations, the digital IR sensor 108, 10-bit pulse widthmodulation (PWM) is configured to continuously transmit the measuredtemperature in range of −20 . . . 120° C., with an output resolution of0.14° C. The factory default power on reset (POR) setting is SMBus.

In some implementations, the digital IR sensor 108 is packaged in anindustry standard TO-39 package.

In some implementations, the generated object and ambient temperaturesare available in RAM of the digital IR sensor 108 with resolution of0.01° C. The temperatures are accessible by 2 wire serial SMBuscompatible protocol (0.02° C. resolution) or via 10-bit PWM (Pulse WidthModulated) output of the digital IR sensor 108.

In some implementations, the digital IR sensor 108 is factory calibratedin wide temperature ranges: −40 . . . 85° C. for the ambient temperatureand −70 . . . 380° C. for the object temperature.

In some implementations of the digital IR sensor 108, the measured valueis the average temperature of all objects in the Field Of View (FOV) ofthe sensor. In some implementations, the digital IR sensor 108 has astandard accuracy of ±0.5° C. around room temperatures, and in someimplementations, the digital IR sensor 108 has an accuracy of ±0.2° C.in a limited temperature range around the human body temperature.

These accuracies are only guaranteed and achievable when the sensor isin thermal equilibrium and under isothermal conditions (there are notemperature differences across the sensor package). The accuracy of thethermometer can be influenced by temperature differences in the packageinduced by causes like (among others): Hot electronics behind thesensor, heaters/coolers behind or beside the sensor or by a hot/coldobject very close to the sensor that not only heats the sensing elementin the thermometer but also the thermometer package. In someimplementations of the digital IR sensor 108, the thermal gradients aremeasured internally and the measured temperature is compensated inconsideration of the thermal gradients, but the effect is not totallyeliminated. It is therefore important to avoid the causes of thermalgradients as much as possible or to shield the sensor from the thermalgradients.

In some implementations, the digital IR sensor 108 is calibrated for anobject emissivity of 1, but in some implementations, the digital IRsensor 108 is calibrated for any emissivity in the range 0.1 . . . 1.0without the need of recalibration with a black body.

In some implementations of the digital IR sensor 108, the PWM can beeasily customized for virtually any range desired by the customer bychanging the content of 2 EEPROM cells. Changing the content of 2 EEPROMcells has no effect on the factory calibration of the device. The PWMpin can also be configured to act as a thermal relay (input is To), thusallowing for an easy and cost effective implementation in thermostats ortemperature (freezing/boiling) alert applications. The temperaturethreshold is programmable by the microprocessor 102 of the non-touchthermometer. In a non-touch thermometer having a SMBus system theprogramming can act as a processor interrupt that can trigger readingall slaves on the bus and to determine the precise condition.

In some implementations, the digital IR sensor 108 has an optical filter(long-wave pass) that cuts off the visible and near infra-red radiantflux is integrated in the package to provide ambient and sunlightimmunity. The wavelength pass band of the optical filter is from 5.5till 14 μm.

In some implementations, the digital IR sensor 108 is controlled by aninternal state machine, which controls the measurements and generationsof the object and ambient temperatures and does the post-processing ofthe temperatures to output the temperatures through the PWM output orthe SMBus compatible interface.

Some implementations of the non-touch thermometer includes 2 IR sensors,the output of the IR sensors being amplified by a low noise low offsetchopper amplifier with programmable gain, converted by a Sigma Deltamodulator to a single bit stream and fed to a DSP for furtherprocessing. The signal is treated by programmable (by means of EEPROMcontend) FIR and IIR low pass filters for further reduction of thebandwidth of the input signal to achieve the desired noise performanceand refresh rate. The output of the IIR filter is the measurement resultand is available in the internal RAM. 3 different cells are available:One for the on-board temperature sensor and 2 for the IR sensors. Basedon results of the above measurements, the corresponding ambienttemperature Ta and object temperatures To are generated. Both generatedtemperatures have a resolution of 0.01° C. The data for Ta and To isread in two ways: Reading RAM cells dedicated for this purpose via the2-wire interface (0.02° C. resolution, fixed ranges), or through the PWMdigital output (10 bit resolution, configurable range). In the last stepof the measurement cycle, the measured Ta and To are rescaled to thedesired output resolution of the PWM) and the regenerated data is loadedin the registers of the PWM state machine, which creates a constantfrequency with a duty cycle representing the measured data.

In some implementations, the digital IR sensor 108 includes a SCL pinfor Serial clock input for 2 wire communications protocol, whichsupports digital input only, used as the clock for SMBus compatiblecommunication. The SCL pin has the auxiliary function for building anexternal voltage regulator. When the external voltage regulator is used,the 2-wire protocol for a power supply regulator is overdriven.

In some implementations, the digital IR sensor 108 includes a slavedeviceA/PWM pin for Digital input/output. In normal mode the measuredobject temperature is accessed at this pin Pulse Width Modulated. InSMBus compatible mode the pin is automatically configured as open drainNMOS. Digital input/output, used for both the PWM output of the measuredobject temperature(s) or the digital input/output for the SMBus. In PWMmode the pin can be programmed in EEPROM to operate as Push/Pull or opendrain NMOS (open drain NMOS is factory default). In SMBus mode slavedeviceA is forced to open drain NMOS I/O, push-pull selection bitdefines PWM/Thermal relay operation. The PWM/slave deviceA pin thedigital IR sensor 108 operates as PWM output, depending on the EEPROMsettings. When WPWM is enabled, after POR the PWM/slave deviceA pin isdirectly configured as PWM output. When the digital IR sensor 108 is inPWM mode, SMBus communication is restored by a special command. In someimplementations, the digital IR sensor 108 is read via PWM or SMBuscompatible interface. Selection of PWM output is done in EEPROMconfiguration (factory default is SMBus). PWM output has twoprogrammable formats, single and dual data transmission, providingsingle wire reading of two temperatures (dual zone object or object andambient). The PWM period is derived from the on-chip oscillator and isprogrammable.

In some implementations, the digital IR sensor 108 includes a VDD pinfor External supply voltage and a VSS pin for ground.

The microprocessor 102 has read access to the RAM and EEPROM and writeaccess to 9 EEPROM cells (at addresses 0x00, 0x01, 0x02, 0x03, 0x04,0x05*, 0x0E, 0x0F, 0x09). When the access to the digital IR sensor 108is a read operation, the digital IR sensor 108 responds with 16 databits and 8 bit PEC only if its own slave address, programmed in internalEEPROM, is equal to the SA, sent by the master. A slave feature allowsconnecting up to 127 devices (SA=0x00 . . . 0x07F) with only 2 wires. Inorder to provide access to any device or to assign an address to a slavedevice before slave device is connected to the bus system, thecommunication starts with zero slave address followed by low R/W bit.When the zero slave address followed by low R/W bit sent from themicroprocessor 102, the digital IR sensor 108 responds and ignores theinternal chip code information.

In some implementations, two digital IR sensors 108 are not configuredwith the same slave address on the same bus.

In regards to bus protocol, after every received 8 bits the slave deviceshould issue ACK or NACK. When a microprocessor 102 initiatescommunication, the microprocessor 102 first sends the address of theslave and only the slave device which recognizes the address will ACK,the rest will remain silent. In case the slave device NACKs one of thebytes, the microprocessor 102 stops the communication and repeat themessage. A NACK could be received after the packet error code (PEC). ANACK after the PEC means that there is an error in the received messageand the microprocessor 102 will try resending the message. PECgeneration includes all bits except the START, REPEATED START, STOP,ACK, and NACK bits. The PEC is a CRC-8 with polynomial X8+X2+X1+1. TheMost Significant Bit of every byte is transferred first.

In single PWM output mode the settings for PWM1 data only are used. Thetemperature reading can be generated from the signal timing as:

$T_{OUT} = {\left( {\frac{2\; t_{2}}{T} \times \left( {T_{O\_ MAX} - T_{O\_ MIN}} \right)} \right) + T_{O\_ MIN}}$

where Tmin and Tmax are the corresponding rescale coefficients in EEPROMfor the selected temperature output (Ta, object temperature range isvalid for both Tobj1 and Tobj2 as specified in the previous table) and Tis the PWM period. Tout is TO1, TO2 or Ta according to Config Register[5:4] settings.

The different time intervals t1 . . . t4 have following meaning:

t1: Start buffer. During t1 the signal is always high. t1=0.125 s×T(where T is the PWM period)

t2: Valid Data Output Band, 0 . . . ½T. PWM output data resolution is 10bit.

t3: Error band—information for fatal error in EEPROM (double errordetected, not correctable).

t3=0.25 s×T. Therefore a PWM pulse train with a duty cycle of 0.875 willindicate a fatal error in EEPROM (for single PWM format). FE means FatalError.

In regards to a format for extended PWM, the temperature transmitted inData 1 field can be generated using the following equation:

$T_{{OUT}\; 1} = {\left( {\frac{4\; t_{2}}{T} \times \left( {T_{{MAX}\; 1} - T_{{MIN}\; 1}} \right)} \right) + T_{{MIN}\; 1}}$

For Data 2 field the equation is:

$T_{{OUT}\; 2} = {\left( {\frac{4\; t_{5}}{T} \times \left( {T_{{MAX}\; 2} - T_{{MIN}\; 2}} \right)} \right) + T_{{MIN}\; 2}}$

FIG. 3 is a block diagram of a non-touch thermometer 300 having a colordisplay device, according to an implementation. In FIG. 3, the displaydevice 114 of FIG. 1 is a LED color display device.

In regards to the structural relationship of the digital infrared sensor108 and the microprocessor 102 in FIG. 1-3, heat radiation on thedigital infrared sensor 108 from any source such as the microprocessor102 or heat sink, will distort detection of infrared energy by thedigital infrared sensor 108. In order to prevent or at least reduce heattransfer between the digital infrared sensor 108 and the microprocessor102, the non-touch thermometers 100, 200 and 300 are low-powered devicesand thus low heat-generating devices that are also powered by a battery104; and that are only used for approximately a 5 second period of timefor each measurement (1 second to acquire the temperature samples andgenerate the body core temperature result, and 4 seconds to display thatresult to the operator) so there is little heat generated by thenon-touch thermometers 100, 200 and 300 in active use.

The internal layout of the non-touch thermometers 100, 200 and 300minimizes as practically as possible the digital infrared sensor as faraway in distance from all other components such the microprocessor 102within the practical limitations of the industrial design of thenon-touch thermometers 100, 200 and 300.

More specifically, to prevent or at least reduce heat transfer betweenthe digital infrared sensor 108 and the microprocessor 102, the digitalinfrared sensor 108 is isolated on a separate PCB from the PCB that hasthe microprocessor 102, as shown in FIG. 23, and the two PCBs areconnected by only a connector that has 4 pins. The minimal connection ofthe single connector having 4 pins reduces heat transfer from themicroprocessor 102 to the digital infrared sensor 108 through theelectrical connector and through transfer that would occur through thePCB material if the digital infrared sensor 108 and the microprocessor102 were mounted on the same PCB.

Digital Infrared Sensor Method Implementations

In the previous section, apparatus of the operation of an implementationwas described. In this section, the particular methods performed bynon-touch thermometer 100, 200 and 300 of such an implementation aredescribed by reference to a series of flowcharts.

FIG. 4 is a flowchart of a method 400 to determine a temperature from adigital infrared sensor, according to an implementation. Method 400includes receiving from the digital readout ports of a digital infraredsensor a digital signal that is representative of an infrared signaldetected by the digital infrared sensor, at block 402.

Method 400 also includes determining a temperature from the digitalsignal that is representative of the infrared signal, at block 404.

FIG. 5 is a flowchart of a method 500 to display temperature colorindicators, according to an implementation of three colors. Method 500provides color rendering in the color LED 2312 to indicate a generalrange of a temperature.

Method 500 includes receiving a temperature (such as temperature 120 inFIG. 1), at block 501.

Method 500 also includes determining whether or not the temperature isin the range of 32.0° C. and 37.3° C., at block 502. If the temperatureis in the range of 32.0° C. and 37.3° C., then the color is set to‘amber’ to indicate a temperature that is low, at block 504 and thebackground of the color LED 2312 is activated in accordance with thecolor, at block 506.

If the temperature is not the range of 32.0° C. and 37.3° C., thenmethod 500 also includes determining whether or not the temperature isin the range of 37.4° C. and 38.0° C., at block 508. If the sensedtemperature is in the range of 37.4° C. and 38.0° C., then the color isset to green to indicate no medical concern, at block 510 and thebackground of the color LED 2312 is activated in accordance with thecolor, at block 506.

If the temperature is not the range of 37.4° C. and 38.0° C., thenmethod 500 also includes determining whether or not the temperature isover 38.0° C., at block 512. If the temperature is over 38.0° C., thenthe color is set to ‘red’ to indicate alert, at block 512 and thebackground of the color LED 2312 is activated in accordance with thecolor, at block 506.

Method 500 assumes that temperature is in gradients of 10ths of adegree. Other temperature range boundaries are used in accordance withother gradients of temperature sensing.

In some implementations, some pixels in the color LED 2312 are activatedas an amber color when the temperature is between 36.3° C. and 37.3° C.(97.3° F. to 99.1° F.), some pixels in the color LED 2312 are activatedas a green when the temperature is between 37.4° C. and 37.9° C. (99.3°F. to 100.2° F.), some pixels in the color LED 2312 are activated as ared color when the temperature is greater than 38° C. (100.4° F.). Insome implementations, the color LED 2312 is a backlit LCD screen 302 inFIG. 3 (which is easy to read in a dark room) and some pixels in thecolor LED 2312 are activated (remain lit) for about 5 seconds after thesingle button 106 is released. After the color LED 2312 has shut off,another temperature reading can be taken by the apparatus. The colorchange of the color LED 2312 is to alert the operator of the apparatusof a potential change of body temperature of the human or animalsubject. The temperature reported on the display can be used fortreatment decisions.

FIG. 6 is a flowchart of a method 600 to manage power in a non-touchdevice having a digital infrared sensor, according to an implementation.The method 600 manages power in the device, such as non-touchthermometer in FIG. 1-3, the non-touch thermometer 2400 in FIG. 24, thehand-held device 2100 in FIG. 21 and/or the computer 2200 in FIG. 22 inorder to reduce heat pollution in the digital infrared sensor.

To prevent or at least reduce heat transfer between the digital infraredsensor 108 and the microprocessor 102, microprocessor 2404 In FIG. 24,main processor 2102 in FIG. 21 or processing unit 2204 in FIG. 22, thecomponents of the non-touch thermometers 100, 200 and 300 in FIG. 1-3,the non-touch thermometer 2400 in FIG. 24, the hand-held device 2100 inFIG. 21 and/or the computer 2200 in FIG. 22 are power controlled, i.e.the non-touch thermometers 100, 200 and 300 in FIG. 1-3, the non-touchthermometer 2400 in FIG. 24, the hand-held device 2100 in FIG. 21 and/orthe computer 2200 in FIG. 22 turn subsystems on and off, and thecomponents are only activated when needed in the measurement and displayprocess, which reduces power consumption and thus heat generation by themicroprocessor 102, microprocessor 2404 In FIG. 24, main processor 2102in FIG. 21 or processing unit 2204 in FIG. 22, of the non-touchthermometers 100, 200 and 300 in FIG. 1-3, the non-touch thermometer2400 in FIG. 24, the hand-held device 2100 in FIG. 21 and/or thecomputer 2200 in FIG. 22, respectively. When not in use, at block 602,the non-touch thermometers 100, 200 and 300 in FIG. 1-3, the non-touchthermometer 2400 in FIG. 24, the hand-held device 2100 in FIG. 21 and/orthe computer 2200 in FIG. 22 are completely powered-off, at block 604(including the main PCB having the microprocessor 102, microprocessor2404 In FIG. 24, main processor 2102 in FIG. 21 or processing unit 2204in FIG. 22, and the sensor PCB having the digital infrared sensor 108)and not drawing any power, other than a power supply, i.e. a boostregulator, which has the effect that the non-touch thermometers 100, 200and 300 in FIG. 1-3, the non-touch thermometer 2400 in FIG. 24, thehand-held device 2100 in FIG. 21 and/or the computer 2200 in FIG. 22draw only drawing micro-amps from the battery 104 while in the offstate, which is required for the life time requirement of 3 years ofoperation, but which also means that in the non-use state there is verylittle powered circuitry in the non-touch thermometers 100, 200 and 300in FIG. 1-3, the non-touch thermometer 2400 in FIG. 24, the hand-helddevice 2100 in FIG. 21 and/or the computer 2200 in FIG. 22 and thereforevery little heat generated in the non-touch thermometers 100, 200 and300 in FIG. 1-3, the non-touch thermometer 2400 in FIG. 24, thehand-held device 2100 in FIG. 21 and/or the computer 2200 in FIG. 22.

When the non-touch thermometers 100, 200 and 300 in FIG. 1-3, thenon-touch thermometer 2400 in FIG. 24, the hand-held device 2100 in FIG.21 and/or the computer 2200 in FIG. 22 are started by the operator, atblock 606, only the microprocessor 102, microprocessor 2404 In FIG. 24,main processor 2102 in FIG. 21 or processing unit 2204 in FIG. 22,digital infrared sensor 108, and low power LCD (e.g. display device 114)are turned on for the first 1 second, at block 608, to take thetemperature measurement via the digital infrared sensor 108 and generatethe body core temperature result via the microprocessor 102 in FIG. 1-3,microprocessor 2404 in FIG. 24, main processor 2102 in FIG. 21 orprocessing unit 2204 in FIG. 22, at block 610. In this way, the mainheat generating components (the LCD 114, the main PCB having themicroprocessor 102 and the sensor PCB having the digital infrared sensor108), the display back-light and the temperature range indicator (i.e.the traffic light indicator 2312) are not on and therefore notgenerating heat during the critical start-up and measurement process, nomore than 1 second. After the measurement process of block 610 has beencompleted, the digital infrared sensor 108 is turned off, at block 612,to reduce current usage from the batteries and heat generation, and alsothe display back-light and temperature range indicators are turned on,at block 614.

The measurement result is displayed for 4 seconds, at block 616, andthen the non-touch thermometers 100, 200 and 300 in FIG. 1-3, thenon-touch thermometer 2400 in FIG. 24, the hand-held device 2100 in FIG.21 and/or the computer 2200 in FIG. 22 are put in low power-off state,at block 618.

In some implementations of methods and apparatus of FIG. 1-6 an operatorcan take the temperature of a subject at multiple locations on a patientand from the temperatures at multiple locations to determine thetemperature at a number of other locations of the subject. The multiplesource points of which the electromagnetic energy is sensed are mutuallyexclusive to the location of the correlated temperature. In one example,the carotid artery source point on the subject and a forehead sourcepoint are mutually exclusive to the core temperature of the subject, anaxillary temperature of the subject, a rectal temperature of the subjectand an oral temperature of the subject.

The correlation of action can include a calculation based on Formula 1:

T _(body) =|f _(stb)(T _(surface temp) +f _(ntc)(T_(ntc)))+F4_(body)|  Formula 1

-   -   where T_(body) is the temperature of a body or subject    -   where f_(stb) is a mathematical formula of a surface of a body    -   where f_(ntc) is mathematical formula for ambient temperature        reading    -   where T_(surface temp) is a surface temperature determined from        the sensing.    -   where T_(ntc) is an ambient air temperature reading    -   where F4_(body) is a calibration difference in axillary mode,        which is stored or set in a memory of the apparatus either        during manufacturing or in the field. The apparatus also sets,        stores and retrieves F4_(oral), F4_(core), and F4_(rectal) in        the memory.    -   f_(ntc)(T_(ntc)) is a bias in consideration of the temperature        sensing mode. For example f_(axillary)(T_(axillary))=0.2° C.,        f_(oral)(T_(oral))=0.4° C., f_(rectal)(T_(rectal))=0.5° C. and        f_(core)(T_(core))=0.3° C.

In some implementations of determining a correlated body temperature ofcarotid artery by biasing a sensed temperature of a carotid artery, thesensed temperature is biased by +0.5° C. to yield the correlated bodytemperature. In another example, the sensed temperature is biased by−0.5° C. to yield the correlated body temperature. An example ofcorrelating body temperature of a carotid artery follows:

f _(ntc)(T _(ntc))=0.2° C. when T _(ntc)=26.2° C. as retrieved from adata table for body sensing mode.

assumption: T _(surface temp)=37.8° C.

T _(surface temp) +f _(ntc)(T _(ntc))=37.8° C.+0.2° C.=38.0° C.

f _(stb)(T _(surface temp) +f _(ntc)(T _(ntc)))=38° C.+1.4° C.=39.4° C.

assumption: F4_(body)=0.5° C.

T _(body) =|f _(stb)(T _(surface temp) +f _(ntc)(T_(ntc)))+F4_(body)|=139.4° C.+0.5 C|=39.9° C.

The correlated temperature for the carotid artery is 40.0° C.

In an example of correlating temperature of a plurality of externallocations, such as a forehead and a carotid artery to an axillarytemperature, first a forehead temperature is calculated using formula 1as follows:

f _(ntc)(T _(ntc))=0.2° C. when T _(ntc)=26.2° C. as retrieved from adata table for axillary sensing mode.

assumption: T _(surface temp)=37.8° C.

T _(surface temp) +f _(ntc)(T _(ntc))=37.8° C.+0.2° C.=38.0° C.

f _(stb)(T _(surface temp) +f _(ntc)(T _(ntc)))=38° C.+1.4° C.=39.4° C.

assumption: F4_(body)=0° C.

T _(body) =|f _(stb)(T _(surface temp) +f _(ntc)(T _(ntc)))+F4b_(ody)|=139.4° C.+0 C|=39.4° C.

And second, a carotid temperature is calculated using formula 1 asfollows:

f _(ntc)(T _(ntc))=0.6° C. when T _(ntc)=26.4° C. as retrieved from adata table.

assumption: T _(surface temp)=38.0° C.

T _(surface temp) +f _(ntc)(T _(ntc))=38.0° C.+0.6° C.=38.6° C.

f _(stb)(T _(surface temp) +f _(ntc)(T _(ntc)))=38.6° C.+1.4 C=40.0° C.

assumption: F4_(body)=0° C.

T _(body) =|f _(stb)(T _(surface temp) +f _(ntc)(T_(ntc)))+F4_(body)|=|40.0° C.+0 C|=40.0° C.

Thereafter the correlated temperature for the forehead (39.4° C.) andthe correlated temperature for the carotid artery (40.0° C.) areaveraged, yielding the final result of the scan of the forehead and thecarotid artery as 39.7° C.

Vital Sign Motion Amplification Apparatus Implementations

Apparatus in FIG. 7-15 use spatial and temporal signal processing togenerate vital signs from a series of digital images.

FIG. 7 is a block diagram of an apparatus 700 of motion amplification,according to an implementation. Apparatus 700 analyzes the temporal andspatial variations in digital images of an animal subject in order togenerate and communicate biological vital signs.

In some implementations, apparatus 700 includes a skin-pixel-identifier702 that identifies pixel values that are representative of the skin intwo or more images 704. In some implementations the images 704 areframes of a video. The skin-pixel-identifier 702 performs block 1602 inFIG. 16. Some implementations of the skin-pixel-identifier 702 performan automatic seed point based clustering process on the two or moreimages 704. In some implementations, apparatus 700 includes a frequencyfilter 706 that receives the output of the skin-pixel-identifier 702 andapplies a frequency filter to the output of the skin-pixel-identifier702. The frequency filter 706 performs block 1604 in FIG. 16 to processthe images 704 in the frequency domain. In implementations where theapparatus in FIG. 7-15 or the methods in FIG. 16-20 are implemented onnon-touch thermometers 100, 200 or 300 in FIG. 1-3, the images 704 inFIG. 7-15 are the images 124 in FIG. 1-3. In some implementations theapparatus in FIG. 7-15 or the methods in FIG. 16-20 are implemented onthe hand-held device 2100 in FIG. 21.

In some implementations, apparatus 700 includes a regional facialclusterial module 708 that applies spatial clustering to the output ofthe frequency filter 706. The regional facial clusterial module 708performs block 1606 in FIG. 16. In some implementations the regionalfacial clusterial module 708 includes fuzzy clustering, k-meansclustering, expectation-maximization process, Ward's apparatus or seedpoint based clustering.

In some implementations, apparatus 700 includes a frequency-filter 710that applies a frequency filter to the output of the regional facialclusterial module 708. The frequency-filter 710 performs block 1608 inFIG. 16. In some implementations, the frequency-filter 710 is aone-dimensional spatial Fourier Transform, a high pass filter, a lowpass filter, a bandpass filter or a weighted bandpass filter. Someimplementations of frequency-filter 710 includes de-noising (e.g.smoothing of the data with a Gaussian filter). The skin-pixel-identifier702, the frequency filter 706, the regional facial clusterial module 708and the frequency-filter 710 amplify temporal variations (as atemporal-variation-amplifier) in the two or more images 704.

In some implementations, apparatus 700 includes a temporal-variationidentifier 712 that identifies temporal variation of the output of thefrequency-filter 710. Thus, the temporal variation represents temporalvariation of the images 704. The temporal-variation identifier 712performs block 1610 in FIG. 16.

In some implementations, apparatus 700 includes a vital-sign generator714 that generates one or more vital sign(s) 716 from the temporalvariation. The vital sign(s) 716 are displayed for review by ahealthcare worker or stored in a volatile or nonvolatile memory forlater analysis, or transmitted to other devices for analysis.

Fuzzy clustering is a class of processes for cluster analysis in whichthe allocation of data points to clusters is not “hard” (all-or-nothing)but “fuzzy” in the same sense as fuzzy logic. Fuzzy logic being a formof many-valued logic which with reasoning that is approximate ratherthan fixed and exact. In fuzzy clustering, every point has a degree ofbelonging to clusters, as in fuzzy logic, rather than belongingcompletely to just one cluster. Thus, points on the edge of a cluster,may be in the cluster to a lesser degree than points in the center ofcluster. An overview and comparison of different fuzzy clusteringprocesses is available. Any point x has a set of coefficients giving thedegree of being in the kth cluster w_(k)(x). With fuzzy c-means, thecentroid of a cluster is the mean of all points, weighted by a degree ofbelonging of each point to the cluster:

$c_{k} = {\frac{\Sigma_{x}{w_{k}(x)}^{m}x}{\Sigma_{x}{w_{k}(x)}^{m}}.}$

The degree of belonging, w_(k)(x), is related inversely to the distancefrom x to the cluster center as calculated on the previous pass. Thedegree of belonging, w_(k)(x) also depends on a parameter m thatcontrols how much weight is given to the closest center.

k-means clustering is a process of vector quantization, originally fromsignal processing, that is popular for cluster analysis in data mining.k-means clustering partitions n observations into k clusters in whicheach observation belongs to the cluster with the nearest mean, servingas a prototype of the cluster. This results in a partitioning of thedata space into Voronoi cells. A Voronoi Cell being a region within aVoronoi Diagram that is a set of points which is specified beforehand. AVoronoi Diagram being a way of dividing space into a number of regions,k-means clustering uses cluster centers to model the data and tends tofind clusters of comparable spatial extent, like K-means clustering, buteach data point has a fuzzy degree of belonging to each separatecluster.

An expectation-maximization process is an iterative process for findingmaximum likelihood or maximum a posteriori (MAP) estimates of parametersin statistical models, where the model depends on unobserved latentvariables. The expectation-maximization iteration alternates betweenperforming an expectation step, which creates a function for theexpectation of the log-likelihood evaluated using the current estimatefor the parameters, and a maximization step, which computes parametersmaximizing the expected log-likelihood found on the expectation step.These parameter-estimates are then used to determine the distribution ofthe latent variables in the next expectation step.

The expectation maximization process seeks to find the maximizationlikelihood expectation of the marginal likelihood by iterativelyapplying the following two steps:

1. Expectation step (E step): Calculate the expected value of the loglikelihood function, with respect to the conditional distribution of Zgiven X under the current estimate of the parameters θ^((t)):

Q(θ|θ^((t)))=E _(Z|X,θ) _((t)) [log L(θ;X,Z)]

2. Maximization step (M step): Find the parameter that maximizes thisquantity:

$\theta^{({t + 1})} = {\underset{\theta}{argmax}{Q\left( \theta \middle| \theta^{(t)} \right)}}$

Note that in typical models to which expectation maximization isapplied:

1. The observed data points X may be discrete (taking values in a finiteor countably infinite set) or continuous (taking values in anuncountably infinite set). There may in fact be a vector of observationsassociated with each data point.

2. The missing values (aka latent variables) Z are discrete, drawn froma fixed number of values, and there is one latent variable per observeddata point.

3. The parameters are continuous, and are of two kinds: Parameters thatare associated with all data points, and parameters associated with aparticular value of a latent variable (i.e. associated with all datapoints whose corresponding latent variable has a particular value).

The Fourier Transform is an important image processing tool which isused to decompose an image into its sine and cosine components. Theoutput of the transformation represents the image in the Fourier orfrequency domain, while the input image is the spatial domainequivalent. In the Fourier domain image, each point represents aparticular frequency contained in the spatial domain image.

The Discrete Fourier Transform is the sampled Fourier Transform andtherefore does not contain all frequencies forming an image, but only aset of samples which is large enough to fully describe the spatialdomain image. The number of frequencies corresponds to the number ofpixels in the spatial domain image, i.e. the image in the spatial andFourier domain are of the same size.

For a square image of size N×N, the two-dimensional DFT is given by:

${F\left( {k,l} \right)} = {\sum\limits_{i = 0}^{N - 1}\; {\sum\limits_{j = 0}^{N - 1}\; {{f\left( {i,j} \right)}^{- {{2\pi}{({\frac{ki}{N} + \frac{i,j}{N}})}}}}}}$

where f(a,b) is the image in the spatial domain and the exponential termis the basis function corresponding to each point F(k,l) in the Fourierspace. The equation can be interpreted as: the value of each pointF(k,l) is obtained by multiplying the spatial image with thecorresponding base function and summing the result.

The basis functions are sine and cosine waves with increasingfrequencies, i.e. F(0,0) represents the DC-component of the image whichcorresponds to the average brightness and F(N−1,N−1) represents thehighest frequency.

A high-pass filter (HPF) is an electronic filter that passeshigh-frequency signals but attenuates (reduces the amplitude of) signalswith frequencies lower than the cutoff frequency. The actual amount ofattenuation for each frequency varies from filter to filter. A high-passfilter is usually modeled as a linear time-invariant system. A high-passfilter can also be used in conjunction with a low-pass filter to make abandpass filter. The simple first-order electronic high-pass filter isimplemented by placing an input voltage across the series combination ofa capacitor and a resistor and using the voltage across the resistor asan output. The product of the resistance and capacitance (R×C) is thetime constant (t); the product is inversely proportional to the cutofffrequency f_(c), that is:

${f_{c} = {\frac{1}{2\pi \; r} = \frac{1}{2\pi \; {RC}}}},$

where f_(c) is in hertz, r is in seconds, R is in ohms, and C is infarads.

A low-pass filter is a filter that passes low-frequency signals andattenuates (reduces the amplitude of) signals with frequencies higherthan the cutoff frequency. The actual amount of attenuation for eachfrequency varies depending on specific filter design. Low-pass filtersare also known as high-cut filter, or treble cut filter in audioapplications. A low-pass filter is the opposite of a high-pass filter.Low-pass filters provide a smoother form of a signal, removing theshort-term fluctuations, and leaving the longer-term trend. One simplelow-pass filter circuit consists of a resistor in series with a load,and a capacitor in parallel with the load. The capacitor exhibitsreactance, and blocks low-frequency signals, forcing the low-frequencysignals through the load instead. At higher frequencies the reactancedrops, and the capacitor effectively functions as a short circuit. Thecombination of resistance and capacitance gives the time constant of thefilter. The break frequency, also called the turnover frequency orcutoff frequency (in hertz), is determined by the time constant.

A band-pass filter is a device that passes frequencies within a certainrange and attenuates frequencies outside that range. These filters canalso be created by combining a low-pass filter with a high-pass filter.Bandpass is an adjective that describes a type of filter or filteringprocess; bandpass is distinguished from passband, which refers to theactual portion of affected spectrum. Hence, a dual bandpass filter hastwo passbands. A bandpass signal is a signal containing a band offrequencies not adjacent to zero frequency, such as a signal that comesout of a bandpass filter.

FIG. 8 is a block diagram of an apparatus 800 of motion amplification,according to an implementation. Apparatus 800 analyzes the temporal andspatial variations in digital images of an animal subject in order togenerate and communicate biological vital signs.

In some implementations, apparatus 800 includes a skin-pixel-identifier702 that identifies pixel values that are representative of the skin intwo or more images 704. The skin-pixel-identifier 702 performs block1602 in FIG. 16. Some implementations of the skin-pixel-identifier 702performs an automatic seed point based clustering process on the leasttwo images 704.

In some implementations, apparatus 800 includes a frequency filter 706that receives the output of the skin-pixel-identifier 702 and applies afrequency filter to the output of the skin-pixel-identifier 702. Thefrequency filter 706 performs block 1604 in FIG. 16 to process theimages 704 in the frequency domain.

In some implementations, apparatus 800 includes a regional facialclusterial module 708 that applies spatial clustering to the output ofthe frequency filter 706. The regional facial clusterial module 708performs block 1606 in FIG. 16. In some implementations the regionalfacial clusterial module 708 includes fuzzy clustering, k-meansclustering, expectation-maximization process, Ward's apparatus or seedpoint based clustering.

In some implementations, apparatus 800 includes a frequency-filter 710that applies a frequency filter to the output of the regional facialclusterial module 708, to generate a temporal variation. Thefrequency-filter 710 performs block 1608 in FIG. 16. In someimplementations, the frequency-filter 710 is a one-dimensional spatialFourier Transform, a high pass filter, a low pass filter, a bandpassfilter or a weighted bandpass filter. Some implementations offrequency-filter 710 includes de-noising (e.g. smoothing of the datawith a Gaussian filter). The skin-pixel-identifier 702, the frequencyfilter 706, the regional facial clusterial module 708 and thefrequency-filter 710 amplify temporal variations in the two or moreimages 704.

In some implementations, apparatus 800 includes a vital-sign generator714 that generates one or more vital sign(s) 716 from the temporalvariation. The vital sign(s) 716 are displayed for review by ahealthcare worker or stored in a volatile or nonvolatile memory forlater analysis, or transmitted to other devices for analysis.

FIG. 9 is a block diagram of an apparatus 900 of motion amplification,according to an implementation. Apparatus 900 analyzes the temporal andspatial variations in digital images of an animal subject in order togenerate and communicate biological vital signs.

In some implementations, apparatus 900 includes a skin-pixel-identifier702 that identifies pixel values that are representative of the skin intwo or more images 704. The skin-pixel-identifier 702 performs block1602 in FIG. 16. Some implementations of the skin-pixel-identifier 702performs an automatic seed point based clustering process on the leasttwo images 704.

In some implementations, apparatus 900 includes a spatial bandpassfilter 902 that receives the output of the skin-pixel-identifier 702 andapplies a spatial bandpass filter to the output of theskin-pixel-identifier 702. The spatial bandpass filter 902 performsblock 1802 in FIG. 18 to process the images 704 in the spatial domain.

In some implementations, apparatus 900 includes a regional facialclusterial module 708 that applies spatial clustering to the output ofthe frequency filter 706. The regional facial clusterial module 708performs block 1804 in FIG. 18. In some implementations the regionalfacial clusterial module 708 includes fuzzy clustering, k-meansclustering, expectation-maximization process, Ward's apparatus or seedpoint based clustering.

In some implementations, apparatus 900 includes a temporal bandpassfilter 904 that applies a frequency filter to the output of the regionalfacial clusterial module 708. The temporal bandpass filter 904 performsblock 1806 in FIG. 18. In some implementations, the temporal bandpassfilter 904 is a one-dimensional spatial Fourier Transform, a high passfilter, a low pass filter, a bandpass filter or a weighted bandpassfilter. Some implementations of temporal bandpass filter 904 includesde-noising (e.g. smoothing of the data with a Gaussian filter).

The skin-pixel-identifier 702, the spatial bandpass filter 902, theregional facial clusterial module 708 and the temporal bandpass filter904 amplify temporal variations in the two or more images 704.

In some implementations, apparatus 900 includes a temporal-variationidentifier 712 that identifies temporal variation of the output of thefrequency-filter 710. Thus, the temporal variation represents temporalvariation of the images 704. The temporal-variation identifier 712performs block 1808 in FIG. 18.

In some implementations, apparatus 900 includes a vital-sign generator714 that generates one or more vital sign(s) 716 from the temporalvariation. The vital sign(s) 716 are displayed for review by ahealthcare worker or stored in a volatile or nonvolatile memory forlater analysis, or transmitted to other devices for analysis.

FIG. 10 is a block diagram of an apparatus 1000 of motion amplification,according to an implementation.

In some implementations, apparatus 1000 includes a pixel-examiner 1002that examines pixel values of two or more images 704. The pixel-examiner1002 performs block 1902 in FIG. 19.

In some implementations, apparatus 1000 includes a temporal variationdeterminer 1006 that determines a temporal variation of examined pixelvalues. The temporal variation determiner 1006 performs block 1904 inFIG. 19.

In some implementations, apparatus 1000 includes a signal-processor 1008that applies signal processing to the pixel value temporal variation,generating an amplified temporal variation. The signal-processor 1008performs block 1906 in FIG. 19. The signal processing amplifies thetemporal variation, even when the temporal variation is small. In someimplementations, the signal processing performed by signal-processor1008 is temporal bandpass filtering that analyzes frequencies over time.In some implementations, the signal processing performed bysignal-processor 1008 is spatial processing that removes noise.Apparatus 1000 amplifies only small temporal variations in thesignal-processing module.

In some implementations, apparatus 900 includes a vital-sign generator714 that generates one or more vital sign(s) 716 from the temporalvariation. The vital sign(s) 716 are displayed for review by ahealthcare worker or stored in a volatile or nonvolatile memory forlater analysis, or transmitted to other devices for analysis.

While apparatus 1000 can process large temporal variations, an advantagein apparatus 1000 is provided for small temporal variations. Thereforeapparatus 1000 is most effective when the two or more images 704 havesmall temporal variations between the two or more images 704. In someimplementations, a vital sign is generated from the amplified temporalvariations of the two or more images 704 from the signal-processor 1008.

FIG. 11 is a block diagram of an apparatus 1100 of motion amplification,according to an implementation. Apparatus 1100 analyzes the temporal andspatial variations in digital images of an animal subject in order togenerate and communicate biological vital signs.

In some implementations, apparatus 1100 includes askin-pixel-identification module 1102 that identifies pixel values 1106that are representative of the skin in two or more images 1104. Theskin-pixel-identification module 1102 performs block 1602 in FIG. 16.Some implementations of the skin-pixel-identification module 1102perform an automatic seed point based clustering process on the leasttwo images 1104.

In some implementations, apparatus 1100 includes a frequency-filtermodule 1108 that receives the identified pixel values 1106 that arerepresentative of the skin and applies a frequency filter to theidentified pixel values 1106. The frequency-filter module 1108 performsblock 1604 in FIG. 16 to process the images 704 in the frequency domain.Each of the images 704 is Fourier transformed, multiplied with a filterfunction and then re-transformed into the spatial domain. Frequencyfiltering is based on the Fourier Transform. The operator takes an image704 and a filter function in the Fourier domain. The image 704 is thenmultiplied with the filter function in a pixel-by-pixel fashion usingthe formula:

G(k,l)=F(k,l)H(k,l)

where F(k,l) is the input image 704 of identified pixel values 1106 inthe Fourier domain, H(k,l) the filter function and G(k,l) is thefiltered image 1110. To obtain the resulting image in the spatialdomain, G(k,l) is re-transformed using the inverse Fourier Transform. Insome implementations, the frequency-filter module 1108 is atwo-dimensional spatial Fourier Transform, a high pass filter, a lowpass filter, a bandpass filter or a weighted bandpass filter.

In some implementations, apparatus 1100 includes a spatial-clustermodule 1112 that applies spatial clustering to the frequency filteredidentified pixel values of skin 1110, generating spatial clusteredfrequency filtered identified pixel values of skin 1114. Thespatial-cluster module 1112 performs block 1606 in FIG. 16. In someimplementations the spatial-cluster module 1112 includes fuzzyclustering, k-means clustering, expectation-maximization process, Ward'sapparatus or seed point based clustering.

In some implementations, apparatus 1100 includes a frequency-filtermodule 1116 that applies a frequency filter to the spatial clusteredfrequency filtered identified pixel values of skin 1114, which generatesfrequency filtered spatial clustered frequency filtered identified pixelvalues of skin 1118. The frequency-filter module 1116 performs block1608 in FIG. 16. In some implementations, the frequency-filter module1116 is a one-dimensional spatial Fourier Transform, a high pass filter,a low pass filter, a bandpass filter or a weighted bandpass filter. Someimplementations of frequency-filter module 1116 includes de-noising(e.g. smoothing of the data with a Gaussian filter).

The skin-pixel-identification module 1102, the frequency-filter module1108, the spatial-cluster module 1112 and the frequency-filter module1116 amplify temporal variations in the two or more images 704.

In some implementations, apparatus 1100 includes a temporal-variationmodule 1120 that determines temporal variation 1122 of the frequencyfiltered spatial clustered frequency filtered identified pixel values ofskin 1118. Thus, temporal variation 1122 represents temporal variationof the images 704. The temporal-variation module 1120 performs block1610 in FIG. 16.

FIG. 12 is a block diagram of an apparatus 1200 to generate and presentany one of a number of biological vital signs from amplified motion,according to an implementation.

In some implementations, apparatus 1200 includes a blood-flow-analyzermodule 1202 that analyzes a temporal variation to generate a pattern offlow of blood 1204. One example of the temporal variation is temporalvariation 1122 in FIG. 11. In some implementations, the pattern flow ofblood 1204 is generated from motion changes in the pixels and thetemporal variation of color changes in the skin of the images 704. Insome implementations, apparatus 1200 includes a blood-flow displaymodule 1206 that displays the pattern of flow of blood 1204 for reviewby a healthcare worker.

In some implementations, apparatus 1200 includes a heartrate-analyzermodule 1208 that analyzes the temporal variation to generate a heartrate1210. In some implementations, the heartrate 1210 is generated from thefrequency spectrum of the temporal signal in a frequency range for heartbeats, such as (0-10 Hertz). In some implementations, apparatus 1200includes a heartrate display module 1212 that displays the heartrate1210 for review by a healthcare worker.

In some implementations, apparatus 1200 includes a respiratoryrate-analyzer module 1214 that analyzes the temporal variation todetermine a respiratory rate 1216. In some implementations, therespiratory rate 1216 is generated from the motion of the pixels in afrequency range for respiration (0-5 Hertz). In some implementations,apparatus 1200 includes respiratory rate display module 1218 thatdisplays the respiratory rate 1216 for review by a healthcare worker.

In some implementations, apparatus 1200 includes a blood-pressureanalyzer module 1220 that analyzes the temporal variation to a generateblood pressure 1222. In some implementations, the blood-pressureanalyzer module 1220 generates the blood pressure 1222 by analyzing themotion of the pixels and the color changes based on a clustering processand potentially temporal data. In some implementations, apparatus 1200includes a blood pressure display module 1224 that displays the bloodpressure 1222 for review by a healthcare worker.

In some implementations, apparatus 1200 includes an EKG analyzer module1226 that analyzes the temporal variation to generate an EKG 1228. Insome implementations, apparatus 1200 includes an EKG display module 1230that displays the EKG 1228 for review by a healthcare worker.

In some implementations, apparatus 1200 includes a pulse oximetryanalyzer module 1232 that analyzes the temporal variation to generatepulse oximetry 1234. In some implementations, the pulse oximetryanalyzer module 1232 generates the pulse oximetry 1234 by analyzing thetemporal color changes based in conjunction with the k-means clusteringprocess and potentially temporal data. In some implementations,apparatus 1200 includes a pulse oximetry display module 1236 thatdisplays the pulse oximetry 1234 for review by a healthcare worker.

FIG. 13 is a block diagram of an apparatus 1300 of motion amplification,according to an implementation. Apparatus 1300 analyzes the temporal andspatial variations in digital images of an animal subject in order togenerate and communicate biological vital signs.

In some implementations, apparatus 1300 includes askin-pixel-identification module 1102 that identifies pixel values 1106that are representative of the skin in two or more images 704. Theskin-pixel-identification module 1102 performs block 1602 in FIG. 16.Some implementations of the skin-pixel-identification module 1102perform an automatic seed point based clustering process on the leasttwo images 704.

In some implementations, apparatus 1300 includes a frequency-filtermodule 1108 that receives the identified pixel values 1106 that arerepresentative of the skin and applies a frequency filter to theidentified pixel values 1106. The frequency-filter module 1108 performsblock 1604 in FIG. 16 to process the images 704 in the frequency domain.Each of the images 704 is Fourier transformed, multiplied with a filterfunction and then re-transformed into the spatial domain. Frequencyfiltering is based on the Fourier Transform. The operator takes an image704 and a filter function in the Fourier domain. The image 704 is thenmultiplied with the filter function in a pixel-by-pixel fashion usingthe

formula:G(k,l)=F(k,l)H(k,l)

where F(k,l) is the input image 704 of identified pixel values 1106 inthe Fourier domain, H(k,l) the filter function and G(k,l) is thefiltered image 1110. To obtain the resulting image in the spatialdomain, G(k,l) is re-transformed using the inverse Fourier Transform. Insome implementations, the frequency-filter module 1108 is atwo-dimensional spatial Fourier Transform, a high pass filter, a lowpass filter, a bandpass filter or a weighted bandpass filter.

In some implementations, apparatus 1300 includes a spatial-clustermodule 1112 that applies spatial clustering to the frequency filteredidentified pixel values of skin 1110, generating spatial clusteredfrequency filtered identified pixel values of skin 1114. Thespatial-cluster module 1112 performs block 1606 in FIG. 16. In someimplementations the spatial clustering includes fuzzy clustering,k-means clustering, expectation-maximization process, Ward's apparatusor seed point based clustering.

In some implementations, apparatus 1300 includes a frequency-filtermodule 1116 that applies a frequency filter to the spatial clusteredfrequency filtered identified pixel values of skin 1114, which generatesfrequency filtered spatial clustered frequency filtered identified pixelvalues of skin 1118. The frequency-filter module 1116 performs block1608 in FIG. 16 to generate a temporal variation 1122. In someimplementations, the frequency-filter module 1116 is a one-dimensionalspatial Fourier Transform, a high pass filter, a low pass filter, abandpass filter or a weighted bandpass filter. Some implementations ofthe frequency-filter module 1116 includes de-noising (e.g. smoothing ofthe data with a Gaussian filter). The skin-pixel-identification module1102, the frequency-filter module 1108, the spatial-cluster module 1112and the frequency-filter module 1116 amplify temporal variations in thetwo or more images 704.

The frequency-filter module 1116 is operably coupled to one of moremodules in FIG. 12 to generate and present any one or a number ofbiological vital signs from amplified motion in the temporal variation1122.

FIG. 14 is a block diagram of an apparatus 1400 of motion amplification,according to an implementation. Apparatus 1400 analyzes the temporal andspatial variations in digital images of an animal subject in order togenerate and communicate biological vital signs.

In some implementations, apparatus 1400 includes askin-pixel-identification module 1102 that identifies pixel values 1106that are representative of the skin in two or more images 704. Theskin-pixel-identification module 1102 performs block 1602 in FIG. 18.Some implementations of the skin-pixel-identification module 1102perform an automatic seed point based clustering process on the leasttwo images 704. In some implementations, apparatus 1400 includes aspatial bandpass filter module 1402 that applies a spatial bandpassfilter to the identified pixel values 1106, generating spatialbandpassed filtered identified pixel values of skin 1404. In someimplementations, the spatial bandpass filter module 1402 includes atwo-dimensional spatial Fourier Transform, a high pass filter, a lowpass filter, a bandpass filter or a weighted bandpass filter. Thespatial bandpass filter module 1402 performs block 1802 in FIG. 18.

In some implementations, apparatus 1400 includes a spatial-clustermodule 1112 that applies spatial clustering to the frequency filteredidentified pixel values of skin 1110, generating spatial clusteredspatial bandpassed identified pixel values of skin 1406. In someimplementations the spatial clustering includes fuzzy clustering,k-means clustering, expectation-maximization process, Ward's apparatusor seed point based clustering. The spatial-cluster module 1112 performsblock 1804 in FIG. 18.

In some implementations, apparatus 1400 includes a temporal bandpassfilter module 1408 that applies a temporal bandpass filter to thespatial clustered spatial bandpass filtered identified pixel values ofskin 1406, generating temporal bandpass filtered spatial clusteredspatial bandpass filtered identified pixel values of skin 1410. In someimplementations, the temporal bandpass filter is a one-dimensionalspatial Fourier Transform, a high pass filter, a low pass filter, abandpass filter or a weighted bandpass filter. The temporal bandpassfilter module 1408 performs block 1806 in FIG. 18.

In some implementations, apparatus 1400 includes a temporal-variationmodule 1120 that determines temporal variation 1522 of the temporalbandpass filtered spatial clustered spatial bandpass filtered identifiedpixel values of skin 1410. Thus, temporal variation 1522 representstemporal variation of the images 704. The temporal-variation module 1520performs block 1808 of FIG. 18. The temporal-variation module 1520 isoperably coupled to one or more modules in FIG. 12 to generate andpresent any one of a number of biological vital signs from amplifiedmotion in the temporal variation 1522.

FIG. 15 is a block diagram of an apparatus 1500 of motion amplification,according to an implementation.

In some implementations, apparatus 1500 includes apixel-examination-module 1502 that examines pixel values of two or moreimages 704, generating examined pixel values 1504. Thepixel-examination-module 1502 performs block 1902 in FIG. 19.

In some implementations, apparatus 1500 includes a temporal variationdeterminer module 1506 that determines a temporal variation 1508 of theexamined pixel values 1504. The temporal variation determiner module1506 performs block 1904 in FIG. 19.

In some implementations, apparatus 1500 includes a signal-processingmodule 1510 that applies signal processing to the pixel value temporalvariations 1508, generating an amplified temporal variation 1522. Thesignal-processing module 1510 performs block 1906 in FIG. 19. The signalprocessing amplifies the temporal variation 1508, even when the temporalvariation 1508 is small. In some implementations, the signal processingperformed by signal-processing module 1510 is temporal bandpassfiltering that analyzes frequencies over time. In some implementations,the signal processing performed by signal-processing module 1510 isspatial processing that removes noise. Apparatus 1500 amplifies onlysmall temporal variations in the signal-processing module.

While apparatus 1500 can process large temporal variations, an advantagein apparatus 1500 is provided for small temporal variations. Thereforeapparatus 1500 is most effective when the two or more images 704 havesmall temporal variations between the two or more images 704. In someimplementations, a vital sign is generated from the amplified temporalvariations of the two or more images 704 from the signal-processingmodule 1510.

Vital Sign Motion Amplification Method Implementations

FIG. 16-20 each use spatial and temporal signal processing to generatevital signs from a series of digital images.

FIG. 16 is a flowchart of a method 1600 of motion amplification,according to an implementation. Method 1600 analyzes the temporal andspatial variations in digital images of an animal subject in order togenerate and communicate biological vital signs.

In some implementations, method 1600 includes identifying pixel valuesof two or more images that are representative of the skin, at block1602. Some implementations of identifying pixel values that arerepresentative of the skin includes performing an automatic seed pointbased clustering process on the least two images.

In some implementations, method 1600 includes applying a frequencyfilter to the identified pixel values that are representative of theskin, at block 1604. In some implementations, the frequency filter inblock 1604 is a two-dimensional spatial Fourier Transform, a high passfilter, a low pass filter, a bandpass filter or a weighted bandpassfilter.

In some implementations, method 1600 includes applying spatialclustering to the frequency filtered identified pixel values of skin, atblock 1606. In some implementations the spatial clustering includesfuzzy clustering, k-means clustering, expectation-maximization process,Ward's method or seed point based clustering.

In some implementations, method 1600 includes applying a frequencyfilter to the spatial clustered frequency filtered identified pixelvalues of skin, at block 1608. In some implementations, the frequencyfilter in block 1608 is a one-dimensional spatial Fourier Transform, ahigh pass filter, a low pass filter, a bandpass filter or a weightedbandpass filter. Some implementations of applying a frequency filter atblock 1608 include de-noising (e.g. smoothing of the data with aGaussian filter).

Actions 1602, 1604, 1606 and 1608 amplify temporal variations in the twoor more images.

In some implementations, method 1600 includes determining temporalvariation of the frequency filtered spatial clustered frequency filteredidentified pixel values of skin, at block 1610.

In some implementations, method 1600 includes analyzing the temporalvariation to generate a pattern of flow of blood, at block 1612. In someimplementations, the pattern flow of blood is generated from motionchanges in the pixels and the temporal variation of color changes in theskin. In some implementations, method 1600 includes displaying thepattern of flow of blood for review by a healthcare worker, at block1613.

In some implementations, method 1600 includes analyzing the temporalvariation to generate heartrate, at block 1614. In some implementations,the heartrate is generated from the frequency spectrum of the temporalvariation in a frequency range for heart beats, such as (0-10 Hertz). Insome implementations, method 1600 includes displaying the heartrate forreview by a healthcare worker, at block 1615.

In some implementations, method 1600 includes analyzing the temporalvariation to determine respiratory rate, at block 1616. In someimplementations, the respiratory rate is generated from the motion ofthe pixels in a frequency range for respiration (0-5 Hertz). In someimplementations, method 1600 includes displaying the respiratory ratefor review by a healthcare worker, at block 1617.

In some implementations, method 1600 includes analyzing the temporalvariation to generate blood pressure, at block 1618. In someimplementations, the blood pressure is generated by analyzing the motionof the pixels and the color changes based on the clustering process andpotentially temporal data from the infrared sensor. In someimplementations, method 1600 includes displaying the blood pressure forreview by a healthcare worker, at block 1619.

In some implementations, method 1600 includes analyzing the temporalvariation to generate EKG, at block 1620. In some implementations,method 1600 includes displaying the EKG for review by a healthcareworker, at block 1621.

In some implementations, method 1600 includes analyzing the temporalvariation to generate pulse oximetry, at block 1622. In someimplementations, the pulse oximetry is generated by analyzing thetemporal color changes based in conjunction with the k-means clusteringprocess and potentially temporal data from the infrared sensor. In someimplementations, method 1600 includes displaying the pulse oximetry forreview by a healthcare worker, at block 1623.

FIG. 17 is a flowchart of a method of motion amplification, according toan implementation that does not include a separate action of determininga temporal variation. Method 1700 analyzes the temporal and spatialvariations in digital images of an animal subject in order to generateand communicate biological vital signs.

In some implementations, method 1700 includes identifying pixel valuesof two or more images that are representative of the skin, at block1602. Some implementations of identifying pixel values that arerepresentative of the skin includes performing an automatic seed pointbased clustering process on the least two images.

In some implementations, method 1700 includes applying a frequencyfilter to the identified pixel values that are representative of theskin, at block 1604. In some implementations, the frequency filter inblock 1604 is a two-dimensional spatial Fourier Transform, a high passfilter, a low pass filter, a bandpass filter or a weighted bandpassfilter.

In some implementations, method 1700 includes applying spatialclustering to the frequency filtered identified pixel values of skin, atblock 1606. In some implementations the spatial clustering includesfuzzy clustering, k-means clustering, expectation-maximization process,Ward's method or seed point based clustering.

In some implementations, method 1700 includes applying a frequencyfilter to the spatial clustered frequency filtered identified pixelvalues of skin, at block 1608, yielding a temporal variation. In someimplementations, the frequency filter in block 1608 is a one-dimensionalspatial Fourier Transform, a high pass filter, a low pass filter, abandpass filter or a weighted bandpass filter.

In some implementations, method 1700 includes analyzing the temporalvariation to generate a pattern of flow of blood, at block 1612. In someimplementations, the pattern flow of blood is generated from motionchanges in the pixels and the temporal variation of color changes in theskin. In some implementations, method 1700 includes displaying thepattern of flow of blood for review by a healthcare worker, at block1613.

In some implementations, method 1700 includes analyzing the temporalvariation to generate heartrate, at block 1614. In some implementations,the heartrate is generated from the frequency spectrum of the temporalvariation in a frequency range for heart beats, such as (0-10 Hertz). Insome implementations, method 1700 includes displaying the heartrate forreview by a healthcare worker, at block 1615.

In some implementations, method 1700 includes analyzing the temporalvariation to determine respiratory rate, at block 1616. In someimplementations, the respiratory rate is generated from the motion ofthe pixels in a frequency range for respiration (0-5 Hertz). In someimplementations, method 1700 includes displaying the respiratory ratefor review by a healthcare worker, at block 1617.

In some implementations, method 1700 includes analyzing the temporalvariation to generate blood pressure, at block 1618. In someimplementations, the blood pressure is generated by analyzing the motionof the pixels and the color changes based on the clustering process andpotentially temporal data from the infrared sensor. In someimplementations, method 1700 includes displaying the blood pressure forreview by a healthcare worker, at block 1619.

In some implementations, method 1700 includes analyzing the temporalvariation to generate EKG, at block 1620. In some implementations,method 1700 includes displaying the EKG for review by a healthcareworker, at block 1621.

In some implementations, method 1700 includes analyzing the temporalvariation to generate pulse oximetry, at block 1622. In someimplementations, the pulse oximetry is generated by analyzing thetemporal color changes based in conjunction with the k-means clusteringprocess and potentially temporal data from the infrared sensor. In someimplementations, method 1700 includes displaying the pulse oximetry forreview by a healthcare worker, at block 1623.

FIG. 18 is a flowchart of a method 1800 of motion amplification fromwhich to generate and communicate biological vital signs, according toan implementation. Method 1800 analyzes the temporal and spatialvariations in digital images of an animal subject in order to generateand communicate the biological vital signs.

In some implementations, method 1800 includes identifying pixel valuesof two or more images that are representative of the skin, at block1602. Some implementations of identifying pixel values that arerepresentative of the skin includes performing an automatic seed pointbased clustering process on the least two images.

In some implementations, method 1800 includes applying a spatialbandpass filter to the identified pixel values, at block 1802. In someimplementations, the spatial filter in block 1802 is a two-dimensionalspatial Fourier Transform, a high pass filter, a low pass filter, abandpass filter or a weighted bandpass filter.

In some implementations, method 1800 includes applying spatialclustering to the spatial bandpass filtered identified pixel values ofskin, at block 1804. In some implementations the spatial clusteringincludes fuzzy clustering, k-means clustering, expectation-maximizationprocess, Ward's method or seed point based clustering.

In some implementations, method 1800 includes applying a temporalbandpass filter to the spatial clustered spatial bandpass filteredidentified pixel values of skin, at block 1806. In some implementations,the temporal bandpass filter in block 1806 is a one-dimensional spatialFourier Transform, a high pass filter, a low pass filter, a bandpassfilter or a weighted bandpass filter.

In some implementations, method 1800 includes determining temporalvariation of the temporal bandpass filtered spatial clustered spatialbandpass filtered identified pixel values of skin, at block 1808.

In some implementations, method 1800 includes analyzing the temporalvariation to generate and visually display a pattern of flow of blood,at block 1612. In some implementations, the pattern flow of blood isgenerated from motion changes in the pixels and the temporal variationof color changes in the skin. In some implementations, method 1800includes displaying the pattern of flow of blood for review by ahealthcare worker, at block 1613.

In some implementations, method 1800 includes analyzing the temporalvariation to generate heartrate, at block 1614. In some implementations,the heartrate is generated from the frequency spectrum of the temporalvariation in a frequency range for heart beats, such as (0-10 Hertz). Insome implementations, method 1800 includes displaying the heartrate forreview by a healthcare worker, at block 1615.

In some implementations, method 1800 includes analyzing the temporalvariation to determine respiratory rate, at block 1616. In someimplementations, the respiratory rate is generated from the motion ofthe pixels in a frequency range for respiration (0-5 Hertz). In someimplementations, method 1800 includes displaying the respiratory ratefor review by a healthcare worker, at block 1617.

In some implementations, method 1800 includes analyzing the temporalvariation to generate blood pressure, at block 1618. In someimplementations, the blood pressure is generated by analyzing the motionof the pixels and the color changes based on the clustering process andpotentially temporal data from the infrared sensor. In someimplementations, method 1800 includes displaying the blood pressure forreview by a healthcare worker, at block 1619.

In some implementations, method 1800 includes analyzing the temporalvariation to generate EKG, at block 1620. In some implementations,method 1800 includes displaying the EKG for review by a healthcareworker, at block 1621.

In some implementations, method 1800 includes analyzing the temporalvariation to generate pulse oximetry, at block 1622. In someimplementations, the pulse oximetry is generated by analyzing thetemporal color changes based in conjunction with the k-means clusteringprocess and potentially temporal data from the infrared sensor. In someimplementations, method 1800 includes displaying the pulse oximetry forreview by a healthcare worker, at block 1623.

FIG. 19 is a flowchart of a method 1900 of motion amplification,according to an implementation. Method 1900 displays the temporalvariations based on temporal variations in videos that are difficult orimpossible to see with the naked eye. Method 1900 applies spatialdecomposition to a video, and applies temporal filtering to the frames.The resulting signal is then amplified to reveal hidden information.Method 1900 can visualize flow of blood filling a face in the video andalso amplify and reveal small motions, and other vital signs such asblood pressure, respiration, EKG and pulse. Method 1900 can execute inreal time to show phenomena occurring at temporal frequencies selectedby the operator. A combination of spatial and temporal processing ofvideos can amplify subtle variations that reveal important aspects ofthe world. Method 1900 considers a time series of color values at anyspatial location (e.g., a pixel) and amplifies variation in a giventemporal frequency band of interest. For example, method 1900 selectsand then amplifies a band of temporal frequencies including plausiblehuman heart rates. The amplification reveals the variation of redness asblood flows through the face. Lower spatial frequencies are temporallyfiltered (spatial pooling) to allow a subtle input signal to rise abovethe camera sensor and quantization noise. The temporal filteringapproach not only amplifies color variation, but can also reveallow-amplitude motion.

Method 1900 can enhance the subtle motions around the chest of abreathing baby. Method 1900 mathematical analysis employs a linearapproximation related to the brightness constancy assumption used inoptical flow formulations. Method 1900 also derives the conditions underwhich the linear approximation holds. The derivation leads to amultiscale approach to magnify motion without feature tracking or motionestimation. Properties of a voxel of fluid are observed, such aspressure and velocity, which evolve over time. Method 1900 studies andamplifies the variation of pixel values over time, in aspatially-multiscale manner. The spatially-multiscale manner to motionmagnification does not explicitly estimate motion, but ratherexaggerates motion by amplifying temporal color changes at fixedpositions. Method 1900 employs differential approximations that form thebasis of optical flow processes. Method 1900 described herein employslocalized spatial pooling and bandpass filtering to extract and revealvisually the signal corresponding to the pulse. The domain analysisallows amplification and visualization of the pulse signal at eachlocation on the face. Asymmetry in facial blood flow can be a symptom ofarterial problems.

Method 1900 described herein makes imperceptible motions visible using amultiscale approach. Method 1900 amplifies small motions, in oneembodiment. Nearly invisible changes in a dynamic environment can berevealed through spatio-temporal processing of standard monocular videosequences. Moreover, for a range of amplification values that issuitable for various applications, explicit motion estimation is notrequired to amplify motion in natural videos. Method 1900 is well suitedto small displacements and lower spatial frequencies. Single frameworkcan amplify both spatial motion and purely temporal changes (e.g., aheart pulse) and can be adjusted to amplify particular temporalfrequencies. A spatial decomposition module decomposes the input videointo different spatial frequency bands, then applies the same temporalfilter to the spatial frequency bands. The outputted filtered spatialbands are then amplified by an amplification factor, added back to theoriginal signal by adders, and collapsed by a reconstruction module togenerate the output video. The temporal filter and amplification factorscan be tuned to support different applications. For example, the systemcan reveal unseen motions of a camera, caused by the flipping mirrorduring a photo burst.

Method 1900 combines spatial and temporal processing to emphasize subtletemporal changes in a video. Method 1900 decomposes the video sequenceinto different spatial frequency bands. These bands might be magnifieddifferently because (a) the bands might exhibit differentsignal-to-noise ratios or (b) the bands might contain spatialfrequencies for which the linear approximation used in motionmagnification does not hold. In the latter case, method 1900 reduces theamplification for these bands to suppress artifacts. When the goal ofspatial processing is to increase temporal signal-to-noise ratio bypooling multiple pixels, the method spatially low-pass filters theframes of the video and downsamples the video frames for computationalefficiency. In the general case, however, method 1900 computes a fullLaplacian pyramid.

Method 1900 then performs temporal processing on each spatial band.Method 1900 considers the time series corresponding to the value of apixel in a frequency band and applies a bandpass filter to extract thefrequency bands of interest. As one example, method 1900 may selectfrequencies within the range of 0.4-4 Hz, corresponding to 24-240 beatsper minute, if the operator wants to magnify a pulse. If method 1900extracts the pulse rate, then method 1900 can employ a narrow frequencyband around that value. The temporal processing is uniform for allspatial levels and for all pixels within each level. Method 1900 thenmultiplies the extracted bandpassed signal by a magnification factor.alpha. The magnification factor .alpha. can be specified by theoperator, and can be attenuated automatically. Method 1900 adds themagnified signal to the original signal and collapses the spatialpyramid to obtain the final output. Since natural videos are spatiallyand temporally smooth, and since the filtering is performed uniformlyover the pixels, the method implicitly maintains spatiotemporalcoherency of the results. The motion magnification amplifies smallmotion without tracking motion. Temporal processing produces motionmagnification, shown using an analysis that relies on the first-orderTaylor series expansions common in optical flow analyses.

Method 1900 begins with a pixel-examination module in the microprocessor102 of the non-touch thermometer 100, 200 or 300 examining pixel valuesof two or more images 704 from the camera 122, at block 1902.

Method 1900 thereafter determines the temporal variation of the examinedpixel values, at block 1904 by a temporal-variation module in themicroprocessor 102.

A signal-processing module in the microprocessor 102 applies signalprocessing to the pixel value temporal variations, at block 1906. Signalprocessing amplifies the determined temporal variations, even when thetemporal variations are small. Method 1900 amplifies only small temporalvariations in the signal-processing module. While method 1900 can beapplied to large temporal variations, an advantage in method 1900 isprovided for small temporal variations. Therefore method 1900 is mosteffective when the input images 704 have small temporal variationsbetween the images 704. In some implementations, the signal processingat block 1906 is temporal bandpass filtering that analyzes frequenciesover time. In some implementations, the signal processing at block 1906is spatial processing that removes noise.

In some implementations, a vital sign is generated from the amplifiedtemporal variations of the input images 704 from the signal processor atblock 1908. Examples of generating a vital signal from a temporalvariation include as in actions 1612, 1614, 1616, 1618, 1620 and 1622 inFIGS. 16, 17 and 18.

FIG. 20 is a flowchart of a method 2000 of motion amplification fromwhich to generate and communicate biological vital signs, according toan implementation. Method 2000 analyzes the temporal and spatialvariations in digital images of an animal subject in order to generateand communicate the biological vital signs.

In some implementations, method 2000 includes cropping at least twoimages to exclude areas that do not include a skin region, at block2002. For example, the excluded area can be a perimeter area around thecenter of each image, so that an outside border area of the image isexcluded. In some implementations of cropping out the border, about 72%of the width and about 72% of the height of each image is cropped out,leaving only 7.8% of the original uncropped image, which eliminatesabout 11/12 of each image and reduces the amount of processing time forthe remainder of the actions in this process by about 12-fold. This oneaction alone at block 2002 in method 2000 can reduce the processing timeof plurality of images 124 in comparison to method 1800 from 4 minutesto 30 seconds, which is of significant difference to the health workerswho used devices that implement method 2000. In some implementations,the remaining area of the image after cropping in a square area and inother implementation the remaining area after cropping is a circulararea. Depending upon the topography and shape of the area in the imagesthat has the most pertinent portion of the imaged subject, differentgeometries and sizes are most beneficial. The action of cropping theimages at block 2002 can be applied at the beginning of methods 1600,1700, 1800 and 1900 in FIGS. 16, 17, 18 and 19, respectively. In otherimplementations of apparatus 700, 800, 900, 1000, 1100, 1200, 1300, 1400and 1500, a cropper module that performs action 2002 is placed at thebeginning of the modules to greatly decrease processing time of theapparatus.

In some implementations, method 2000 includes identifying pixel valuesof the at least two or more cropped images that are representative ofthe skin, at block 2004. Some implementations of identifying pixelvalues that are representative of the skin include performing anautomatic seed point based clustering process on the least two images.

In some implementations, method 2000 includes applying a spatialbandpass filter to the identified pixel values, at block 1802. In someimplementations, the spatial filter in block 1802 is a two-dimensionalspatial Fourier Transform, a high pass filter, a low pass filter, abandpass filter or a weighted bandpass filter.

In some implementations, method 2000 includes applying spatialclustering to the spatial bandpass filtered identified pixel values ofskin, at block 1804. In some implementations the spatial clusteringincludes fuzzy clustering, k-means clustering, expectation-maximizationprocess, Ward's method or seed point based clustering.

In some implementations, method 2000 includes applying a temporalbandpass filter to the spatial clustered spatial bandpass filteredidentified pixel values of skin, at block 1806. In some implementations,the temporal bandpass filter in block 1806 is a one-dimensional spatialFourier Transform, a high pass filter, a low pass filter, a bandpassfilter or a weighted bandpass filter.

In some implementations, method 2000 includes determining temporalvariation of the temporal bandpass filtered spatial clustered spatialbandpass filtered identified pixel values of skin, at block 1808.

In some implementations, method 2000 includes analyzing the temporalvariation to generate and visually display a pattern of flow of blood,at block 1612. In some implementations, the pattern flow of blood isgenerated from motion changes in the pixels and the temporal variationof color changes in the skin. In some implementations, method 2000includes displaying the pattern of flow of blood for review by ahealthcare worker, at block 1613.

In some implementations, method 2000 includes analyzing the temporalvariation to generate heartrate, at block 1614. In some implementations,the heartrate is generated from the frequency spectrum of the temporalvariation in a frequency range for heart beats, such as (0-10 Hertz). Insome implementations, method 2000 includes displaying the heartrate forreview by a healthcare worker, at block 1615.

In some implementations, method 2000 includes analyzing the temporalvariation to determine respiratory rate, at block 1616. In someimplementations, the respiratory rate is generated from the motion ofthe pixels in a frequency range for respiration (0-5 Hertz). In someimplementations, method 2000 includes displaying the respiratory ratefor review by a healthcare worker, at block 1617.

In some implementations, method 2000 includes analyzing the temporalvariation to generate blood pressure, at block 1618. In someimplementations, the blood pressure is generated by analyzing the motionof the pixels and the color changes based on the clustering process andpotentially temporal data from the infrared sensor. In someimplementations, method 2000 includes displaying the blood pressure forreview by a healthcare worker, at block 1619.

In some implementations, method 2000 includes analyzing the temporalvariation to generate EKG, at block 1620. In some implementations,method 2000 includes displaying the EKG for review by a healthcareworker, at block 1621.

In some implementations, method 2000 includes analyzing the temporalvariation to generate pulse oximetry, at block 1622. In someimplementations, the pulse oximetry is generated by analyzing thetemporal color changes based in conjunction with the k-means clusteringprocess and potentially temporal data from the infrared sensor. In someimplementations, method 2000 includes displaying the pulse oximetry forreview by a healthcare worker, at block 1623.

In some implementations, methods 1600-2000 are implemented as a sequenceof instructions which, when executed by a microprocessor 102 in FIG.1-3, microprocessor 2404 In FIG. 24, main processor 2102 in FIG. 21 orprocessing unit 2204 in FIG. 22, cause the processor to perform therespective method. In other implementations, methods 1600-2000 areimplemented as a computer-accessible medium having computer executableinstructions capable of directing a microprocessor, such asmicroprocessor 102 in FIG. 1-3, microprocessor 2404 in FIG. 24, mainprocessor 2102 in FIG. 21 or processing unit 2204 in FIG. 22, to performthe respective method. In different implementations, the medium is amagnetic medium, an electronic medium, or an optical medium.

Hardware and Operating Environment

FIG. 21 is a block diagram of a hand-held device 2100, according to animplementation. The hand-held device 2100 may also have the capabilityto allow voice communication. Depending on the functionality provided bythe hand-held device 2100, the hand-held device 2100 may be referred toas a data messaging device, a two-way pager, a cellular telephone withdata messaging capabilities, a wireless Internet appliance, or a datacommunication device (with or without telephony capabilities).

The hand-held device 2100 includes a number of modules such as a mainprocessor 2102 that controls the overall operation of the hand-helddevice 2100. Communication functions, including data and voicecommunications, are performed through a communication subsystem 2104.The communication subsystem 2104 receives messages from and sendsmessages to wireless networks 2105. In other implementations of thehand-held device 2100, the communication subsystem 2104 can beconfigured in accordance with the Global System for Mobile Communication(GSM), General Packet Radio Services (GPRS), Enhanced Data GSMEnvironment (EDGE), Universal Mobile Telecommunications Service (UMTS),data-centric wireless networks, voice-centric wireless networks, anddual-mode networks that can support both voice and data communicationsover the same physical base stations. Combined dual-mode networksinclude, but are not limited to, Code Division Multiple Access (CDMA) orCDMA2000 networks, GSM/GPRS networks (as mentioned above), and futurethird-generation (3G) networks like EDGE and UMTS. Some other examplesof data-centric networks include Mobitex™ and DataTAC™ networkcommunication systems. Examples of other voice-centric data networksinclude Personal Communication Systems (PCS) networks like GSM and TimeDivision Multiple Access (TDMA) systems.

The wireless link connecting the communication subsystem 2104 with thewireless network 2105 represents one or more different Radio Frequency(RF) channels. With newer network protocols, these channels are capableof supporting both circuit switched voice communications and packetswitched data communications.

The main processor 2102 also interacts with additional subsystems suchas a Random Access Memory (RAM) 2106, a flash memory 2108, a display2110, an auxiliary input/output (I/O) subsystem 2112, a data port 2114,a keyboard 2116, a speaker 2118, a microphone 2120, short-rangecommunications subsystem 2122 and other device subsystems 2124. In someimplementations, the flash memory 2108 includes a hybrid femtocell/Wi-Fiprotocol stack 2109. The stack 2109 supports authentication andauthorization between the hand-held device 2100 into a shared Wi-Finetwork and both a 3G and 4G mobile networks.

Some of the subsystems of the hand-held device 2100 performcommunication-related functions, whereas other subsystems may provide“resident” or on-device functions. By way of example, the display 2110and the keyboard 2116 may be used for both communication-relatedfunctions, such as entering a text message for transmission over thewireless network 2105, and device-resident functions such as acalculator or task list.

The hand-held device 2100 can transmit and receive communication signalsover the wireless network 2105 after required network registration oractivation procedures have been completed. Network access is associatedwith a subscriber or user of the hand-held device 2100. To identify asubscriber, the hand-held device 2100 requires a SIM/RUIM card 2126(i.e. Subscriber Identity Module or a Removable User Identity Module) tobe inserted into a SIM/RUIM interface 2128 in order to communicate witha network. The SIM card or RUIM 2126 is one type of a conventional“smart card” that can be used to identify a subscriber of the hand-helddevice 2100 and to personalize the hand-held device 2100, among otherthings. Without the SIM card 2126, the hand-held device 2100 is notfully operational for communication with the wireless network 2105. Byinserting the SIM card/RUIM 2126 into the SIM/RUIM interface 2128, asubscriber can access all subscribed services. Services may include: webbrowsing and messaging such as e-mail, voice mail, Short Message Service(SMS), and Multimedia Messaging Services (MMS). More advanced servicesmay include: point of sale, field service and sales force automation.The SIM card/RUIM 2126 includes a processor and memory for storinginformation. Once the SIM card/RUIM 2126 is inserted into the SIM/RUIMinterface 2128, the SIM is coupled to the main processor 2102. In orderto identify the subscriber, the SIM card/RUIM 2126 can include some userparameters such as an International Mobile Subscriber Identity (IMSI).An advantage of using the SIM card/RUIM 2126 is that a subscriber is notnecessarily bound by any single physical mobile device. The SIMcard/RUIM 2126 may store additional subscriber information for thehand-held device 2100 as well, including datebook (or calendar)information and recent call information. Alternatively, useridentification information can also be programmed into the flash memory2108.

The hand-held device 2100 is a battery-powered device and includes abattery interface 2132 for receiving one or more rechargeable batteries2130. In one or more implementations, the battery 2130 can be a smartbattery with an embedded microprocessor. The battery interface 2132 iscoupled to a regulator 2133, which assists the battery 2130 in providingpower V+ to the hand-held device 2100. Although current technology makesuse of a battery, future technologies such as micro fuel cells mayprovide the power to the hand-held device 2100.

The hand-held device 2100 also includes an operating system 2134 andmodules 2136 to 2149 which are described in more detail below. Theoperating system 2134 and the modules 2136 to 2149 that are executed bythe main processor 2102 are typically stored in a persistent nonvolatilemedium such as the flash memory 2108, which may alternatively be aread-only memory (ROM) or similar storage element (not shown). Thoseskilled in the art will appreciate that portions of the operating system2134 and the modules 2136 to 2149, such as specific device applications,or parts thereof, may be temporarily loaded into a volatile store suchas the RAM 2106. Other modules can also be included.

The subset of modules 2136 that control basic device operations,including data and voice communication applications, will normally beinstalled on the hand-held device 2100 during its manufacture. Othermodules include a message application 2138 that can be any suitablemodule that allows a user of the hand-held device 2100 to transmit andreceive electronic messages. Various alternatives exist for the messageapplication 2138 as is well known to those skilled in the art. Messagesthat have been sent or received by the user are typically stored in theflash memory 2108 of the hand-held device 2100 or some other suitablestorage element in the hand-held device 2100. In one or moreimplementations, some of the sent and received messages may be storedremotely from the hand-held device 2100 such as in a data store of anassociated host system with which the hand-held device 2100communicates.

The modules can further include a device state module 2140, a PersonalInformation Manager (PIM) 2142, and other suitable modules (not shown).The device state module 2140 provides persistence, i.e. the device statemodule 2140 ensures that important device data is stored in persistentmemory, such as the flash memory 2108, so that the data is not lost whenthe hand-held device 2100 is turned off or loses power.

The PIM 2142 includes functionality for organizing and managing dataitems of interest to the user, such as, but not limited to, e-mail,contacts, calendar events, voice mails, appointments, and task items. APIM application has the ability to transmit and receive data items viathe wireless network 2105. PIM data items may be seamlessly integrated,synchronized, and updated via the wireless network 2105 with thehand-held device 2100 subscriber's corresponding data items storedand/or associated with a host computer system. This functionalitycreates a mirrored host computer on the hand-held device 2100 withrespect to such items. This can be particularly advantageous when thehost computer system is the hand-held device 2100 subscriber's officecomputer system.

The hand-held device 2100 also includes a connect module 2144, and an ITpolicy module 2146. The connect module 2144 implements the communicationprotocols that are required for the hand-held device 2100 to communicatewith the wireless infrastructure and any host system, such as anenterprise system, with which the hand-held device 2100 is authorized tointerface. Examples of a wireless infrastructure and an enterprisesystem are given in FIGS. 21 and 22, which are described in more detailbelow.

The connect module 2144 includes a set of APIs that can be integratedwith the hand-held device 2100 to allow the hand-held device 2100 to useany number of services associated with the enterprise system. Theconnect module 2144 allows the hand-held device 2100 to establish anend-to-end secure, authenticated communication pipe with the hostsystem. A subset of applications for which access is provided by theconnect module 2144 can be used to pass IT policy commands from the hostsystem to the hand-held device 2100. This can be done in a wireless orwired manner. These instructions can then be passed to the IT policymodule 2146 to modify the configuration of the hand-held device 2100.Alternatively, in some cases, the IT policy update can also be done overa wired connection.

The IT policy module 2146 receives IT policy data that encodes the ITpolicy. The IT policy module 2146 then ensures that the IT policy datais authenticated by the hand-held device 2100. The IT policy data canthen be stored in the flash memory 2106 in its native form. After the ITpolicy data is stored, a global notification can be sent by the ITpolicy module 2146 to all of the applications residing on the hand-helddevice 2100. Applications for which the IT policy may be applicable thenrespond by reading the IT policy data to look for IT policy rules thatare applicable.

The IT policy module 2146 can include a parser 2147, which can be usedby the applications to read the IT policy rules. In some cases, anothermodule or application can provide the parser. Grouped IT policy rules,described in more detail below, are retrieved as byte streams, which arethen sent (recursively) into the parser to determine the values of eachIT policy rule defined within the grouped IT policy rule. In one or moreimplementations, the IT policy module 2146 can determine whichapplications are affected by the IT policy data and transmit anotification to only those applications. In either of these cases, forapplications that are not being executed by the main processor 2102 atthe time of the notification, the applications can call the parser orthe IT policy module 2146 when the applications are executed todetermine if there are any relevant IT policy rules in the newlyreceived IT policy data.

All applications that support rules in the IT Policy are coded to knowthe type of data to expect. For example, the value that is set for the“WEP User Name” IT policy rule is known to be a string; therefore thevalue in the IT policy data that corresponds to this rule is interpretedas a string. As another example, the setting for the “Set MaximumPassword Attempts” IT policy rule is known to be an integer, andtherefore the value in the IT policy data that corresponds to this ruleis interpreted as such.

After the IT policy rules have been applied to the applicableapplications or configuration files, the IT policy module 2146 sends anacknowledgement back to the host system to indicate that the IT policydata was received and successfully applied.

The programs 2137 can also include a temporal-variation-amplifier 2148and a vital sign generator 2149. In some implementations, thetemporal-variation-amplifier 2148 includes a skin-pixel-identifier 702,a frequency-filter 706, a regional facial clusterial module 708 and afrequency filter 710 as in FIGS. 7 and 8. In some implementations, thetemporal-variation-amplifier 2148 includes a skin-pixel-identifier 702,a spatial bandpass-filter 902, regional facial clusterial module 708 anda temporal bandpass filter 904 as in FIG. 9. In some implementations,the temporal-variation-amplifier 2148 includes a pixel-examiner 1002, atemporal variation determiner 1006 and signal processor 1008 as in FIG.10. In some implementations, the temporal-variation-amplifier 2148includes a skin-pixel-identification module 1102, a frequency-filtermodule 1108, spatial-cluster module 1112 and a frequency filter module1116 as in FIGS. 11 and 12. In some implementations, thetemporal-variation-amplifier module 1102, a spatial bandpass filtermodule 1402, a spatial-cluster module 1112 and a temporal bandpassfilter module 1406 as in FIG. 14. In some implementations, thetemporal-variation-amplifier 2148 includes a pixel examination-module1502, a temporal variation determiner module 1506 and a signalprocessing module 1510 as in FIG. 15. The camera 122 captures images 124and the vital sign generator 2149 generates the vital sign(s) 716 thatis displayed by display 2110 or transmitted by communication subsystem2104 or short-range communications subsystem 2122, enunciated by speaker2118 or stored by flash memory 2108.

Other types of modules can also be installed on the hand-held device2100. These modules can be third party modules, which are added afterthe manufacture of the hand-held device 2100. Examples of third partyapplications include games, calculators, utilities, etc.

The additional applications can be loaded onto the hand-held device 2100through at least one of the wireless network 2105, the auxiliary I/Osubsystem 2112, the data port 2114, the short-range communicationssubsystem 2122, or any other suitable device subsystem 2124. Thisflexibility in application installation increases the functionality ofthe hand-held device 2100 and may provide enhanced on-device functions,communication-related functions, or both. For example, securecommunication applications may enable electronic commerce functions andother such financial transactions to be performed using the hand-helddevice 2100.

The data port 2114 enables a subscriber to set preferences through anexternal device or module and extends the capabilities of the hand-helddevice 2100 by providing for information or module downloads to thehand-held device 2100 other than through a wireless communicationnetwork. The alternate download path may, for example, be used to loadan encryption key onto the hand-held device 2100 through a direct andthus reliable and trusted connection to provide secure devicecommunication.

The data port 2114 can be any suitable port that enables datacommunication between the hand-held device 2100 and another computingdevice. The data port 2114 can be a serial or a parallel port. In someinstances, the data port 2114 can be a USB port that includes data linesfor data transfer and a supply line that can provide a charging currentto charge the battery 2130 of the hand-held device 2100.

The short-range communications subsystem 2122 provides for communicationbetween the hand-held device 2100 and different systems or devices,without the use of the wireless network 2105. For example, the subsystem2122 may include an infrared device and associated circuits and modulesfor short-range communication. Examples of short-range communicationstandards include standards developed by the Infrared Data Association(IrDA), Bluetooth, and the 802.11 family of standards developed by IEEE.

Bluetooth is a wireless technology standard for exchanging data overshort distances (using short-wavelength radio transmissions in the ISMband from 2400-2480 MHz) from fixed and mobile devices, creatingpersonal area networks (PANs) with high levels of security. Created bytelecom vendor Ericsson in 1994, Bluetooth was originally conceived as awireless alternative to RS-232 data cables. Bluetooth can connectseveral devices, overcoming problems of synchronization. Bluetoothoperates in the range of 2400-2483.5 MHz (including guard bands), whichis in the globally unlicensed Industrial, Scientific and Medical (ISM)2.4 GHz short-range radio frequency band. Bluetooth uses a radiotechnology called frequency-hopping spread spectrum. The transmitteddata is divided into packets and each packet is transmitted on one ofthe 79 designated Bluetooth channels. Each channel has a bandwidth of 1MHz. The first channel starts at 2402 MHz and continues up to 2480 MHzin 1 MHz steps. The first channel usually performs 1600 hops per second,with Adaptive Frequency-Hopping (AFH) enabled. Originally Gaussianfrequency-shift keying (GFSK) modulation was the only modulation schemeavailable; subsequently, since the introduction of Bluetooth 2.0+EDR,π/4-DQPSK and 8DPSK modulation may also be used between compatibledevices. Devices functioning with GFSK are said to be operating in basicrate (BR) mode where an instantaneous data rate of 1 Mbit/s is possible.The term Enhanced Data Rate (EDR) is used to describe π/4-DPSK and 8DPSKschemes, each giving 2 and 3 Mbit/s respectively. The combination ofthese (BR and EDR) modes in Bluetooth radio technology is classified asa “BR/EDR radio”. Bluetooth is a packet-based protocol with amaster-slave structure. One master may communicate with up to 7 slavesin a piconet; all devices share the master's clock. Packet exchange isbased on the basic clock, defined by the master, which ticks at 312.5 μsintervals. Two clock ticks make up a slot of 625 μs; two slots make up aslot pair of 1250 μs. In the simple case of single-slot packets themaster transmits in even slots and receives in odd slots; the slave,conversely, receives in even slots and transmits in odd slots. Packetsmay be 1, 3 or 5 slots long but in all cases the master transmit willbegin in even slots and the slave transmit in odd slots. A masterBluetooth device can communicate with a maximum of seven devices in apiconet (an ad-hoc computer network using Bluetooth technology), thoughnot all devices reach this maximum. The devices can switch roles, byagreement, and the slave can become the master (for example, a headsetinitiating a connection to a phone will necessarily begin as master, asinitiator of the connection; but may subsequently prefer to be slave).The Bluetooth Core Specification provides for the connection of two ormore piconets to form a scatternet, in which certain devicessimultaneously play the master role in one piconet and the slave role inanother. At any given time, data can be transferred between the masterand one other device (except for the little-used broadcast mode. Themaster chooses which slave device to address; typically, the masterswitches rapidly from one device to another in a round-robin fashion.Since the master chooses which slave to address, whereas a slave is (intheory) supposed to listen in each receive slot, being a master is alighter burden than being a slave. Being a master of seven slaves ispossible; being a slave of more than one master is difficult. Many ofthe services offered over Bluetooth can expose private data or allow theconnecting party to control the Bluetooth device. For security reasonsit is necessary to be able to recognize specific devices and thus enablecontrol over which devices are allowed to connect to a given Bluetoothdevice. At the same time, it is useful for Bluetooth devices to be ableto establish a connection without user intervention (for example, assoon as the Bluetooth devices of each other are in range). To resolvethis conflict, Bluetooth uses a process called bonding, and a bond iscreated through a process called pairing. The pairing process istriggered either by a specific request from a user to create a bond (forexample, the user explicitly requests to “Add a Bluetooth device”), orthe pairing process is triggered automatically when connecting to aservice where (for the first time) the identity of a device is requiredfor security purposes. These two cases are referred to as dedicatedbonding and general bonding respectively. Pairing often involves somelevel of user interaction; this user interaction is the basis forconfirming the identity of the devices. Once pairing successfullycompletes, a bond will have been formed between the two devices,enabling those two devices to connect to each other in the futurewithout requiring the pairing process in order to confirm the identityof the devices. When desired, the bonding relationship can later beremoved by the user. Secure Simple Pairing (SSP): This is required byBluetooth v2.1, although a Bluetooth v2.1 device may only use legacypairing to interoperate with a v2.0 or earlier device. Secure SimplePairing uses a form of public key cryptography, and some types can helpprotect against man in the middle, or MITM attacks. SSP has thefollowing characteristics: Just works: As implied by the name, thismethod just works. No user interaction is required; however, a devicemay prompt the user to confirm the pairing process. This method istypically used by headsets with very limited IO capabilities, and ismore secure than the fixed PIN mechanism which is typically used forlegacy pairing by this set of limited devices. This method provides noman in the middle (MITM) protection. Numeric comparison: If both deviceshave a display and at least one can accept a binary Yes/No user input,both devices may use Numeric Comparison. This method displays a 6-digitnumeric code on each device. The user should compare the numbers toensure that the numbers are identical. If the comparison succeeds, theuser(s) should confirm pairing on the device(s) that can accept aninput. This method provides MITM protection, assuming the user confirmson both devices and actually performs the comparison properly. PasskeyEntry: This method may be used between a device with a display and adevice with numeric keypad entry (such as a keyboard), or two deviceswith numeric keypad entry. In the first case, the display is used toshow a 6-digit numeric code to the user, who then enters the code on thekeypad. In the second case, the user of each device enters the same6-digit number. Both of these cases provide MITM protection. Out of band(OOB): This method uses an external means of communication, such as NearField Communication (NFC) to exchange some information used in thepairing process. Pairing is completed using the Bluetooth radio, butrequires information from the OOB mechanism. This provides only thelevel of MITM protection that is present in the OOB mechanism. SSP isconsidered simple for the following reasons: In most cases, SSP does notrequire a user to generate a passkey. For use-cases not requiring MITMprotection, user interaction can be eliminated. For numeric comparison,MITM protection can be achieved with a simple equality comparison by theuser. Using OOB with NFC enables pairing when devices simply get close,rather than requiring a lengthy discovery process.

In use, a received signal such as a text message, an e-mail message, orweb page download will be processed by the communication subsystem 2104and input to the main processor 2102. The main processor 2102 will thenprocess the received signal for output to the display 2110 oralternatively to the auxiliary I/O subsystem 2112. A subscriber may alsocompose data items, such as e-mail messages, for example, using thekeyboard 2116 in conjunction with the display 2110 and possibly theauxiliary I/O subsystem 2112. The auxiliary subsystem 2112 may includedevices such as: a touch screen, mouse, track ball, infrared fingerprintdetector, or a roller wheel with dynamic button pressing capability. Thekeyboard 2116 is preferably an alphanumeric keyboard and/ortelephone-type keypad. However, other types of keyboards may also beused. A composed item may be transmitted over the wireless network 2105through the communication subsystem 2104.

For voice communications, the overall operation of the hand-held device2100 is substantially similar, except that the received signals areoutput to the speaker 2118, and signals for transmission are generatedby the microphone 2120. Alternative voice or audio I/O subsystems, suchas a voice message recording subsystem, can also be implemented on thehand-held device 2100. Although voice or audio signal output isaccomplished primarily through the speaker 2118, the display 2110 canalso be used to provide additional information such as the identity of acalling party, duration of a voice call, or other voice call relatedinformation.

FIG. 22 is a block diagram of a hardware and operating environment 2200in which different implementations can be practiced. The description ofFIG. 22 provides an overview of computer hardware and a suitablecomputing environment in conjunction with which some implementations canbe implemented. Implementations are described in terms of a computerexecuting computer-executable instructions. However, someimplementations can be implemented entirely in computer hardware inwhich the computer-executable instructions are implemented in read-onlymemory. Some implementations can also be implemented in client/servercomputing environments where remote devices that perform tasks arelinked through a communications network. Program modules can be locatedin both local and remote memory storage devices in a distributedcomputing environment.

FIG. 22 illustrates an example of a computer environment 2200 useful inthe context of the environment of FIG. 1-9, in accordance with animplementation. The computer environment 2200 includes a computationresource 2202 capable of implementing the processes described herein. Itwill be appreciated that other devices can alternatively used thatinclude more modules, or fewer modules, than those illustrated in FIG.22.

The illustrated operating environment 2200 is only one example of asuitable operating environment, and the example described with referenceto FIG. 22 is not intended to suggest any limitation as to the scope ofuse or functionality of the implementations of this disclosure. Otherwell-known computing systems, environments, and/or configurations can besuitable for implementation and/or application of the subject matterdisclosed herein.

The computation resource 2202 includes one or more processors orprocessing units 2204, a system memory 2206, and a bus 2208 that couplesvarious system modules including the system memory 2206 to processingunit 2204 and other elements in the environment 2200. The bus 2208represents one or more of any of several types of bus structures,including a memory bus or memory controller, a peripheral bus, anaccelerated graphics port and a processor or local bus using any of avariety of bus architectures, and can be compatible with SCSI (smallcomputer system interconnect), or other conventional bus architecturesand protocols.

The system memory 2206 includes nonvolatile read-only memory (ROM) 2210and random access memory (RAM) 2212, which can or can not includevolatile memory elements. A basic input/output system (BIOS) 2214,containing the elementary routines that help to transfer informationbetween elements within computation resource 2202 and with externalitems, typically invoked into operating memory during start-up, isstored in ROM 2210.

The computation resource 2202 further can include a non-volatileread/write memory 2216, represented in FIG. 22 as a hard disk drive,coupled to bus 2208 via a data media interface 2217 (e.g., a SCSI, ATA,or other type of interface); a magnetic disk drive (not shown) forreading from, and/or writing to, a removable magnetic disk 2220 and anoptical disk drive (not shown) for reading from, and/or writing to, aremovable optical disk 2226 such as a CD, DVD, or other optical media.

The non-volatile read/write memory 2216 and associated computer-readablemedia provide nonvolatile storage of computer-readable instructions,data structures, program modules and other data for the computationresource 2202. Although the exemplary environment 2200 is describedherein as employing a non-volatile read/write memory 2216, a removablemagnetic disk 2220 and a removable optical disk 2226, it will beappreciated by those skilled in the art that other types ofcomputer-readable media which can store data that is accessible by acomputer, such as magnetic cassettes, FLASH memory cards, random accessmemories (RAMs), read only memories (ROM), and the like, can also beused in the exemplary operating environment.

A number of program modules can be stored via the non-volatileread/write memory 2216, magnetic disk 2220, optical disk 2226, ROM 2210,or RAM 2212, including an operating system 2230, one or more applicationprograms 2232, program modules 2234 and program data 2236. Examples ofcomputer operating systems conventionally employed include the NUCLEUS®operating system, the LINUX® operating system, and others, for example,providing capability for supporting application programs 2232 using, forexample, code modules written in the C++® computer programming language.The application programs 2232 and/or the program modules 2234 can alsoinclude a temporal-variation-amplifier (as shown in 2148 in FIG. 21) anda vital sign generator (as shown in 2149 in FIG. 22). In someimplementations, the temporal-variation-amplifier 2148 in theapplication programs 2232 and/or the program modules 2234 includes askin-pixel-identifier 702, a frequency-filter 706, regional facialclusterial module 708 and a frequency filter 710 as in FIGS. 7 and 8. Insome implementations, the temporal-variation-amplifier 2148 inapplication programs 2232 and/or the program modules 2234 includes askin-pixel-identifier 702, a spatial bandpass-filter 902, regionalfacial clusterial module 708 and a temporal bandpass filter 904 as inFIG. 9. In some implementations, the temporal-variation-amplifier 2148in the application programs 2232 and/or the program modules 2234includes a pixel-examiner 1002, a temporal variation determiner 1006 andsignal processor 1008 as in FIG. 10. In some implementations, thetemporal-variation-amplifier 2148 in the application programs 2232and/or the program modules 2234 includes a skin-pixel-identificationmodule 1102, a frequency-filter module 1108, spatial-cluster module 1112and a frequency filter module 1116 as in FIGS. 11 and 12. In someimplementations, the temporal-variation-amplifier 2148 in theapplication programs 2232 and/or the program modules 2234 includes askin-pixel-identification module 1102, a spatial bandpass filter module1402, a spatial-cluster module 1112 and a temporal bandpass filtermodule 1406 as in FIG. 14. In some implementations, thetemporal-variation-amplifier 2148 in the application programs 2232and/or the program modules 2234 includes a pixel examination-module1502, a temporal variation determiner module 1506 and a signalprocessing module 1510 as in FIG. 15. The camera 122 captures images 124that are processed by the temporal-variation-amplifier 2148 and thevital sign generator 2149 to generate the vital sign(s) 716 that isdisplayed by display 2250 or transmitted by computation resource 2202,enunciated by a speaker or stored in program data 2236.

A user can enter commands and information into computation resource 2202through input devices such as input media 2238 (e.g., keyboard/keypad,tactile input or pointing device, mouse, foot-operated switchingapparatus, joystick, touchscreen or touchpad, microphone, antenna etc.).Such input devices 2238 are coupled to the processing unit 2204 througha conventional input/output interface 2242 that is, in turn, coupled tothe system bus. Display 2250 or other type of display device is alsocoupled to the system bus 2208 via an interface, such as a video adapter2252.

The computation resource 2202 can include capability for operating in anetworked environment using logical connections to one or more remotecomputers, such as a remote computer 2260. The remote computer 2260 canbe a personal computer, a server, a router, a network PC, a peer deviceor other common network node, and typically includes many or all of theelements described above relative to the computation resource 2202. In anetworked environment, program modules depicted relative to thecomputation resource 2202, or portions thereof, can be stored in aremote memory storage device such as can be associated with the remotecomputer 2260. By way of example, remote application programs 2262reside on a memory device of the remote computer 2260. The logicalconnections represented in FIG. 22 can include interface capabilities,e.g., such as interface capabilities in FIG. 5, a storage area network(SAN, not illustrated in FIG. 22), local area network (LAN) 2272 and/ora wide area network (WAN) 2274, but can also include other networks.

Such networking environments are commonplace in modern computer systems,and in association with intranets and the Internet. In certainimplementations, the computation resource 2202 executes an Internet Webbrowser program (which can optionally be integrated into the operatingsystem 2230), such as the “Internet Explorer®” Web browser manufacturedand distributed by the Microsoft Corporation of Redmond, Wash.

When used in a LAN-coupled environment, the computation resource 2202communicates with or through the local area network 2272 via a networkinterface or adapter 2276 and typically includes interfaces, such as amodem 2278, or other apparatus, for establishing communications with orthrough the WAN 2274, such as the Internet. The modem 2278, which can beinternal or external, is coupled to the system bus 2208 via a serialport interface.

In a networked environment, program modules depicted relative to thecomputation resource 2202, or portions thereof, can be stored in remotememory apparatus. It will be appreciated that the network connectionsshown are exemplary, and other means of establishing a communicationslink between various computer systems and elements can be used.

A user of a computer can operate in a networked environment usinglogical connections to one or more remote computers, such as a remotecomputer 2260, which can be a personal computer, a server, a router, anetwork PC, a peer device or other common network node. Typically, aremote computer 2260 includes many or all of the elements describedabove relative to the computer 2200 of FIG. 22.

The computation resource 2202 typically includes at least some form ofcomputer-readable media. Computer-readable media can be any availablemedia that can be accessed by the computation resource 2202. By way ofexample, and not limitation, computer-readable media can comprisecomputer storage media and communication media.

Computer storage media include volatile and nonvolatile, removable andnon-removable media, implemented in any method or technology for storageof information, such as computer-readable instructions, data structures,program modules or other data. The term “computer storage media”includes, but is not limited to, RAM, ROM, EEPROM, FLASH memory or othermemory technology, CD, DVD, or other optical storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other media which can be used to storecomputer-intelligible information and which can be accessed by thecomputation resource 2202.

Communication media typically embodies computer-readable instructions,data structures, program modules or other data, represented via, anddeterminable from, a modulated data signal, such as a carrier wave orother transport mechanism, and includes any information delivery media.The term “modulated data signal” means a signal that has one or more ofits characteristics set or changed in such a manner as to encodeinformation in the signal in a fashion amenable to computerinterpretation.

By way of example, and not limitation, communication media include wiredmedia, such as wired network or direct-wired connections, and wirelessmedia, such as acoustic, RF, infrared and other wireless media. Thescope of the term computer-readable media includes combinations of anyof the above.

FIG. 23 is a representation of display 2300 that is presented on thedisplay device of apparatus in FIG. 1-3, according to an implementation.

Some implementations of display 2300 include a representation of threedetection modes 2302, a first detection mode being detection and displayof surface temperature, a second detection mode being detection anddisplay of body temperature and a third detection mode being detectionand display of room temperature.

Some implementations of display 2300 include a representation of Celsius2304 that is activated when the apparatus is in Celsius mode.

Some implementations of display 2300 include a representation of asensed temperature 2306.

Some implementations of display 2300 include a representation ofFahrenheit 2308 that is activated when the apparatus is in Fahrenheitmode.

Some implementations of display 2300 include a representation of a mode2310 of site temperature sensing, a first site mode being detection ofan axillary surface temperature, a second site mode being detection ofan oral temperature, a third site mode being detection of a rectaltemperature and a fourth site mode being detection of a coretemperature.

Some implementations of display 2300 include a representation of atemperature traffic light 2312, in which a green traffic light indicatesthat the temperature 120 is good; an amber traffic light indicates thatthe temperature 120 is low; and a red traffic light indicates that thetemperature 120 is high.

Some implementations of display 2300 include a representation of a probemode 2314 that is activated when the sensed temperature 2306 is from acontact sensor.

Some implementations of display 2300 include a representation of thecurrent time/date 2316 of the apparatus.

FIG. 24-28 are schematics of the electronic components of a non-touchthermometer 2400 having a digital IR sensor. FIG. 24 is a portion of theschematic of the non-touch thermometer 2400 having a digital IR sensor,according to an implementation. As discussed above in regards to FIG. 2and FIG. 3, thermal isolation of the digital IR sensor is an importantfeature. In second circuit board 2401, a digital IR sensor 2403 isthermally isolated from the heat of the microprocessor 2404 (shown inFIG. 25) through a first digital interface 2402. The digital IR sensor2403 is not mounted on the same circuit board 2405 as the microprocessor2404 (shown in FIG. 25) which reduces heat transfer from a first circuitboard 2405 to the digital IR sensor 2403. The non-touch thermometer 2400also includes a second circuit board 2401, the second circuit board 2401including a second digital interface 2412, the second digital interface2412 being operably coupled to the first digital interface 2402 and adigital infrared sensor 2403 being operably coupled to the seconddigital interface 2412, the digital infrared sensor 2403 having portsthat provide only digital readout. The microprocessor 2404 (shown inFIG. 25) is operable to receive from the ports that provide only digitalreadout a digital signal that is representative of an infrared signalgenerated by the digital infrared sensor 2403 and the microprocessor2404 (shown in FIG. 25) is operable to determine a temperature from thedigital signal that is representative of the infrared signal. The firstcircuit board 2405 includes all of the components in FIG. 24, FIG. 25and FIG. 26 other than the second circuit board 2401, the digital IRsensor 2403 and the second digital interface 2412.

FIG. 25 is a portion of the schematic of the non-touch thermometer 2400having the digital IR sensor, according to an implementation. Anon-touch thermometer 2400 includes a first circuit board 2405, thefirst circuit board 2405 including the microprocessor 2404.

FIG. 26 is a portion of the schematic of the non-touch thermometer 2400having the digital IR sensor, according to an implementation. The firstcircuit board 2405 includes a display device that is operably coupled tothe microprocessor 2404 through a display interface 2408.

FIG. 27 is a circuit 2700 that is a portion of the schematic of thenon-touch thermometer 2400 having the digital IR sensor, according to animplementation. Circuit 2700 includes a battery 2406 that is operablycoupled to the microprocessor 2404, a single button 2410 that isoperably coupled to the microprocessor 2404.

FIG. 28 is a circuit 2800 that is a portion of the schematic of thenon-touch thermometer 2400 having the digital IR sensor, according to animplementation.

The non-touch thermometer further includes a housing, and where thebattery 104 is fixedly attached to the housing. The non-touchthermometer where an exterior portion of the housing further includes: amagnet.

CONCLUSION

A non-touch thermometer that senses temperature from a digital infraredsensor is described. A technical effect of the apparatus is transmittingfrom the digital infrared sensor a digital signal representing atemperature without conversion from analog. Another technical effect ofthe apparatus and methods disclosed herein is generating a temporalvariation of images from which a heartrate and the respiratory rate canbe determined and displayed or stored. Although specific implementationsare illustrated and described herein, it will be appreciated by those ofordinary skill in the art that any arrangement which is generated toachieve the same purpose may be substituted for the specificimplementations shown. This application is intended to cover anyadaptations or variations.

In particular, one of skill in the art will readily appreciate that thenames of the methods and apparatus are not intended to limitimplementations. Furthermore, additional methods and apparatus can beadded to the modules, functions can be rearranged among the modules, andnew modules to correspond to future enhancements and physical devicesused in implementations can be introduced without departing from thescope of implementations. One of skill in the art will readily recognizethat implementations are applicable to future non-touch temperaturesensing devices, different temperature measuring sites on humans oranimals and new display devices.

The terminology used in this application meant to include alltemperature sensors, processors and operator environments and alternatetechnologies which provide the same functionality as described herein.

1. An apparatus of motion amplification to communicate biological vitalsigns, the apparatus comprising: a cropper that is operable to receiveat least two images and crop the image to exclude a border area of theimages, generating at least two cropped images; a skin-pixel-identifierthat is operably coupled to the cropper and that identifies pixel valuesthat are representative of the skin in at least two cropped images; afirst frequency filter that is operably coupled to theskin-pixel-identifier and that applies a frequency filter to output ofthe skin-pixel-identifier; a regional facial clusterial module that isoperably coupled to the first frequency filter and that applies spatialclustering to the output of the first frequency filter; a secondfrequency filter that is operably regional facial clusterial module andthat is applied to output of the regional facial clusterial module,generating a temporal variation; a vital-sign generator that is operablycoupled to the second frequency filter that generates at least one vitalsign from the temporal variation; and a display device that is operablycoupled to the vital-sign generator that is operable to display the atleast one vital sign.
 2. The apparatus of claim 1, wherein theskin-pixel-identifier further comprises: an automatic seed point basedclustering apparatus.
 3. The apparatus of claim 1, wherein the regionalfacial clusterial module further comprises: a fuzzy clusterer.
 4. Theapparatus of claim 1, wherein the first frequency filter furthercomprises: a high pass filter.
 5. The apparatus of claim 1, wherein thefirst frequency filter further comprises: a low pass filter.
 6. Theapparatus of claim 1, wherein the first frequency filter furthercomprises: a bandpass filter.
 7. The apparatus of claim 1, wherein thefirst frequency filter further comprises: a weighted bandpass filter. 8.The apparatus of claim 1, wherein the first frequency filter furthercomprises: a Gaussian filter.
 9. The apparatus of claim 1 furthercomprising: a storage device that is operably coupled to the vital-signgenerator and that is operable to store the at least one vital sign in avolatile memory.
 10. The apparatus of claim 1 further comprising: astorage device that is operably coupled to the vital-sign generator andthat is operable to transmit the at least one vital sign to anotherapparatus.
 11. An apparatus of motion amplification to communicatebiological vital signs, the apparatus comprising: a cropper that isoperable to receive at least two images and operable to crop the imagesto exclude a border area of the images, generating at least two croppedimages; a first frequency filter that applies a frequency filter to theat least two cropped images; a regional facial clusterial module that isoperably coupled to the first frequency filter and that applies spatialclustering to output of the first frequency filter; a second frequencyfilter that is operably to the regional facial clusterial module andthat is applied to output of the regional facial clusterial module,generating a temporal variation; a vital-sign generator that is operablycoupled to the second frequency filter that generates at least one vitalsign from the temporal variation; and a display device that is operablycoupled to the vital-sign generator that is operable to display the atleast one vital sign.
 12. The apparatus of claim 11 further comprising:a skin-pixel-identifier that is operably coupled to the first frequencyfilter that identifies pixel values that are representative of skin inthe at least two images.
 13. The apparatus of claim 11, wherein theregional facial clusterial module further comprises: a fuzzy clusterer.14. The apparatus of claim 11, wherein the regional facial clusterialmodule further comprises: a seed point based clustering apparatus. 15.The apparatus of claim 11, wherein the first frequency filter furthercomprises: a one-dimensional spatial Fourier Transformation apparatus.16. The apparatus of claim 11, wherein the first frequency filterfurther comprises: a high pass filter.
 17. The apparatus of claim 11,wherein the first frequency filter further comprises: a low pass filter.18. The apparatus of claim 11, wherein the first frequency filterfurther comprises: a Gaussian filter.
 19. The apparatus of claim 11further comprising: a storage device that is operably coupled to thevital-sign generator and that is operable to store the at least onevital sign in a volatile memory.
 20. The apparatus of claim 11 furthercomprising: a storage device that is operably coupled to the vital-signgenerator and that is operable to transmit the at least one vital signto another apparatus.